The Software Efficiency Report · From the Founder's Desk
The Software Efficiency Report | 2026 Week 43
Database changes are where zero-downtime goals die
This week’s report covers the latest developments across cloud platforms, Linux, DevOps, security, AI agents and embedded systems, with a focus on what matters to engineering teams.
We look at new tools, platform updates, security incidents and industry trends that affect software delivery, infrastructure reliability and operational efficiency.
The deep dive explores why database changes can cause unexpected downtime and how safer migration practices, careful sequencing and proper testing help reduce release risks.
We also cover release safety, database backup recovery and work-in-progress limits, with practical ideas to help teams improve reliability, reduce delivery bottlenecks and ship software with greater confidence.
- Metric of the week
- Forward-Fix Ratio (FFR): Target < 10%
- Deep dive
- Database changes are where zero-downtime goals die
Software Efficiency Metric of the Week
Forward-Fix Ratio (FFR): Target < 10%
Tracks how often your team tries to patch a broken release with another live commit instead of hitting the rollback button.
What it is: A measure of release safety and operational discipline. When a bad deployment breaks production, it checks whether your default muscle memory is an immediate, clean revert or a high-stress scramble to push a hasty hotfix.
Why it matters: Fixing forward under pressure is gambling with uptime. Panic-pushed patches bypass normal review cycles, skip edge-case testing, and routinely trigger secondary, nastier outages. When teams avoid rolling back, it usually isn't out of bravery—it's because they can't. Tightly coupled database schemas, dirty migrations, and messy release pipelines trap them into moving forward, turning a 5-minute blip into a four-hour fire drill.
The Fix:
- Revert first, ask questions later: Make rollbacks the mandatory default for any release-induced Sev-1 or Sev-2 incident. Triage and fix the bug in staging, not in front of users.
- Decouple migrations from code: Enforce the expand-contract pattern for database changes. Schema changes must always remain backward-compatible with the previous app version so a code rollback never corrupts data.
- Build one-click rollbacks: If rolling back requires manual Git surgery or pipeline gymnastics taking longer than 5 minutes, treat that rollback friction as an urgent platform bug.
Formula: FFR (%) = (Incidents resolved via forward-fix hotpatches / Total release-related incidents) × 100
Quick Example: If 10 deployments trigger production alerts this quarter and your team pushes live hotfixes for 7 of them instead of reverting to the last known good build:
FFR = (7 / 10) × 100 = 70%
(A red flag that your release path has no reverse gear and your engineers are debugging live in production).
Reader Poll
When did your team last test restoring a production database from backup, or are you just trusting your cloud console's green checkmark?
My take: A backup you have never restored is just an unverified pile of bytes taking up disk space. Teams usually discover this mid-crisis: the snapshot takes twelve hours to rehydrate, the KMS key was deleted with an old Terraform run, or the engine version mismatch crashes the boot. If you do not test restores on a schedule, your real RTO is whenever your team stops panicking.
How does your team verify production backups actually work?
- A) Automated drills: Scheduled jobs restore snapshots into a sandbox weekly, run query checks, and tear down.
- B) Annual fire drill: We click restore by hand once a year during a nervous maintenance window.
- C) Pure faith: The dashboard says "Snapshot Complete," so we assume it will work in a disaster.
- D) Replicas only: Bad data replicates to every node in milliseconds, so we have no real fallback.
Can you prove you can recover from a database wipe today, or are you just hoping for the best?
Engineering Tip of the Week
Cap active tasks at one per developer to stop sprint stall
When engineers juggle two or three open tickets at once, work crawls. Context-switching between branches eats hours, while half-finished pull requests sit waiting for reviews. Enforcing a strict rule where a developer cannot pick up a new task until their current one is merged or handed off stops inventory from piling up. It forces squads to swarm on code reviews, helping the team finish existing work before starting something new.
Technology Ecosystem Trends
Top Developments/Trends picks for this week Shaping Modern Engineering Efficiency and Operations
Rust keeps colonizing core Linux infrastructure: the upcoming 7.4 kernel pulls in another 17,000 lines of Rust code, a sign the language has moved from experiment to default choice in systems programming even as maintainers argue over how much AI-assisted code belongs in the same tree. Source
CI platforms race to put a leash on coding agents: GitLab added an AI Governance Dashboard tracking every Claude Code, Cursor or Codex session, Chainguard and Buildkite shipped sandboxed workspaces and scoped secrets for agentic CI, and GitHub rolled out a package firewall to screen npm downloads reaching agents. Source Source Source
Git itself becomes the next casualty of agent scale: Anthropic fully migrated off Git to a homegrown system called Scion after its daily merged code changes jumped twelvefold in four months, reporting 200 times less CPU and memory use, though it says Scion isn't fit for anyone else's general use. Source
Kernel contributions grow faster than reviewers can trust them: a new Linux Foundation study estimates a quarter of incoming open-source pull requests are low-quality AI output, and reviewers caught a real race condition in Torvalds' own Claude-Code-assisted kernel patch from last week before it could merge. Source Source
Observability vendors plug straight into the model layer: Anthropic added OpenTelemetry trace export from the Claude Platform itself, with setup guides for a dozen partners including Datadog, Grafana and Honeycomb, turning what used to be scattered third-party instrumentation into a first-party pipeline. Source
AI bug hunters are outrunning patch teams: Bank of America brought in 1,000 extra contractors after Anthropic's Mythos model found vulnerabilities faster than staff could triage them, part of a couple hundred million dollars in disclosed remediation costs, a real price tag on AI-discovered bugs outpacing human review. Source
Agentic AI governance becomes everyone's job at once: OpenAI disclosed four more rogue-agent incidents, bringing its public total to 19, CISA, NSA and NIST jointly published incident-response guidance for autonomous systems, and Microsoft shipped Guardian Agents while joining Nvidia's new Open Agent Safety Platform alongside 100-plus other companies. Source Source Source
Adoption keeps outrunning proof that AI pays for itself: a joint IDC and Lenovo study found 88 percent of AI pilots never reach production, and a separate survey found that although 32 percent of managed service providers have adopted AI, only 12 percent can show measurable payoff from it. Source Source
OT ransomware moves from anecdote to hard numbers: new Armis data finds 73 percent of organizations' building automation systems have already been touched by ransomware, while Dragos tracked industrial ransomware incidents climbing from roughly 1,020 in this year's first quarter to 1,140 in the second. Source
Non-Human Identity (NHI) Governance Becomes Priority Zero: With machine-to-machine connections, CI/CD runners, and AI agent credentials far outnumbering human developers, engineering leaders are deploying intent-based, ephemeral machine access controls to stop service-account privilege sprawl. Source
Tools, Resources and Communities | Worth Knowing
Open Source Tools
- Score: A developer-centric workload specification that translates a single service definition into environments like local Docker Compose or production Kubernetes manifests without exposing engineers to cluster plumbing. Source
- bpftrace: A high-level tracing language for Linux eBPF that allows engineers to inspect kernel events, trace memory allocations, and debug I/O latency on live production servers with one-liners without installing bulky agents or restarting nodes. Source
- Labgrid: An embedded test automation framework written in Python that controls hardware target boards, network power switches, and serial consoles, allowing teams to integrate physical edge devices directly into standard CI/CD pipelines. Source
Commercial Platforms
- Socket: A developer-first security platform that inspects open-source dependencies for proactive supply chain risks like typosquatting, hidden install scripts, and unexpected permission escalation instead of just matching stale CVE databases. Source
- Rootly: An incident management platform native to Slack and Microsoft Teams that automates on-call paging, incident channel setup, timeline capture, and post-mortem generation to cut coordination chaos during live outages. Source
- Torizon: An industrial Linux software platform by Toradex that provides container runtimes, fleet health monitoring, and secure over-the-air updates to bring cloud-native deployment cadences to edge hardware. Source
Learning Resources – Interesting articles
- LWN.net (Linux Weekly News): The authoritative community publication for technical deep dives into Linux kernel development, device driver architecture, patch reviews, and upstream subsystem decisions. Source
- OWASP Top 10 for LLM Applications: A practical security guide documenting the most critical failure modes in generative AI architectures, including prompt injection, training data poisoning, and sensitive information disclosure. Source
- Bootlin Training Materials: A comprehensive, freely accessible collection of lab manuals and slide decks detailing embedded Linux kernel development, device trees, and board support packages. Source
Deep Dive Article : Database changes are where zero-downtime goals die
Application deploys are mostly a solved problem now. Blue-green, canaries, feature flags, automated rollbacks. Plenty of teams ship ten times a day and barely notice.
Then a migration shows up. One line of SQL can undo all of that work.
Here’s why it happens, and what to do about it.
Code rolls back. Data doesn’t.
If a bad build goes out, you redeploy the old one. A few minutes, and it’s forgotten by Friday.
A bad migration is different. Maybe it dropped a column. Maybe a backfill rewrote 80 million rows. Maybe it locked a table that every request touches. Reverting the code fixes none of that.
Database changes are often irreversible. They touch shared state. And they run against production volumes that staging never saw.
In fintech, SaaS and healthcare, the stakes go up again. A locked payments table means failed transactions. A botched migration on patient records means a compliance conversation nobody wants.
The one idea that fixes most of it
During any deploy, old code and new code run at the same time.
Rolling deploys do it. Canaries do it. Even a quick blue-green switch has a window where both versions hit the database.
So every schema change has to work with the code running now and the code about to run. If it only works with one of them, you’ve built an outage. You just haven’t scheduled it yet.
Everything below is a variation on that rule.
Four real incidents worth studying
None of these companies are careless. All four published detailed write-ups, which is why we can learn from them. Not every one is a classic schema migration. That’s part of the lesson. Anything that changes the data layer carries this risk.
Basecamp, November 2018: the migration they put off. Basecamp 3 went read-only for almost five hours. The cause was a primary key column on a very busy events table. It was a regular integer, and it hit its ceiling of 2,147,483,647. The fix was a migration to bigint. The team had been working on it already. They just ran out of runway. In the postmortem, David Heinemeier Hansson called it the worst outage in about a decade, and said it was avoidable. Rails had switched its default to bigint two years earlier.
The takeaway: the migration you keep postponing eventually runs anyway. The only question is whether you pick the time. Check your integer keys on high-volume tables now, while changing them is a calm project instead of an emergency.
Cloudflare, November 2025: the permission change that rewrote a query. On November 18, 2025, a large part of Cloudflare’s network began failing to deliver traffic. The trigger was an access-control change on a ClickHouse cluster. It altered what a metadata query returned, and the query suddenly produced duplicate rows. That query generated a feature file for their Bot Management system. The file doubled in size and went past a limit in the core proxy. Cloudflare called it their worst outage since 2019.
The takeaway: a change doesn’t need to be ALTER TABLE to break things. Permissions, views and metadata can all change what a query returns. Any job that builds something from database output needs to validate that output before it ships. Size, shape, and sanity.
Atlassian, April 2022: the script nobody checked against real data. A script meant to delete a legacy app from customer sites was given site IDs instead of app IDs. It deleted 883 sites, covering 775 customers, in about 23 minutes. Some customers were out for up to 14 days. Atlassian’s review says the script went through normal peer review, but that review focused on the endpoint being called and how, not on the IDs being passed in.
The takeaway: review the data a change touches, not just the code that does it. Dry runs, row counts before and after, and soft deletes would each have shrunk this.
TSB, April 2018: the big-bang cutover. TSB moved about 5.2 million customers to a new platform in one go. Customers were locked out, and it took until December 2018 to get back to normal. The UK regulators later fined the bank £48.65 million for failing to organise and control the programme properly. The bank had paid £32.7 million in redress by then.
The takeaway for regulated teams: poor change control is a regulatory problem as well as an engineering one. Staged, reversible changes are easier to defend to an auditor than one big switch.
Where release outages come from
1. The lock nobody expected. In PostgreSQL, many ALTER TABLE operations need an ACCESS EXCLUSIVE lock. That lock waits behind any running query on the table. While it waits, every new query queues up behind it. One long-running report can turn a one-second migration into a full outage.
2. The table rewrite. Changing a column type can rewrite the whole table. So can adding a column with a volatile default. On a small table, nobody notices. On a 500 GB table, you’ve just scheduled a long maintenance window without telling anyone. Basecamp’s integer-to-bigint change is the classic case.
3. The wrong deploy order. The migration drops a column while old code still reads it. Or new code ships expecting a column that doesn’t exist yet. Both come from treating schema and code as one atomic step. They aren’t.
4. The backfill that eats production. A single UPDATE across a huge table holds locks, bloats the WAL, spikes replica lag and starves real traffic.
5. The ORM surprise. The migration tool generated something nobody read. An index created without CONCURRENTLY. A NOT NULL added in one step. It got through review because it was “just generated.”
6. The migration that can’t be undone. There’s a down script on paper. It was never tested. And it can’t bring back data that’s already gone.
The core pattern: expand and contract
This is the technique that removes most of the risks above. You’ll also see it called parallel change. It turns one risky change into several safe ones.
Say you want to rename customer_name to full_name.
The naive way is one migration that renames the column, plus a code deploy. During the deploy window, half your pods break.
The safe way takes more steps. Each is small and boring, which is the point.
- Expand. Add
full_nameas a nullable column. Nothing breaks, because nothing uses it yet. - Write to both. Deploy code that writes to both columns. Reads still come from the old one.
- Backfill. Copy existing data into the new column in small batches.
- Verify. Compare the two columns and check for drift. Don’t skip this.
- Switch reads. Deploy code that reads the new column. Keep dual writes on.
- Stop writing the old column. Another deploy.
- Contract. Later, once you’re sure, drop the old column.
Seven steps for a rename feels slow. But each step can be deployed, observed and reversed on its own. That’s the real cost of zero-downtime. Not tooling. Patience and sequencing.
Stripe published a well-known write-up on running this kind of migration at scale, with dual writes and careful verification. It’s linked in the references.
Postgres: the details that bite
Set a lock timeout. Always. Start the migration with something like SET lock_timeout = '2s'. If it can’t get the lock quickly, it fails fast and you retry. That’s far better than blocking the table while it waits. Pair it with a retry loop.
Create indexes concurrently. CREATE INDEX CONCURRENTLY avoids blocking writes. It’s slower and can’t run inside a transaction block. Check for invalid indexes afterward, because a failed concurrent build leaves one behind.
Add constraints in two phases. For a foreign key or check constraint, add it as NOT VALID first. That’s quick. Then run VALIDATE CONSTRAINT separately, which takes a much lighter lock.
Get NOT NULL safely. Add a CHECK (col IS NOT NULL) NOT VALID constraint and validate it. Then set the column to NOT NULL. From Postgres 12 onward, that last step skips the full table scan, because the validated check already proves the data is clean.
Know your version. Since Postgres 11, adding a column with a constant default doesn’t rewrite the table. Older versions did.
Keep transactions short. A forgotten open transaction can block a migration for hours. Monitor for idle-in-transaction sessions.
MySQL: a different set of rules
MySQL and MariaDB have online DDL for many operations. Many isn’t all. Whether a change runs in place, copies the table or blocks writes depends on the exact operation and version.
For big tables, most teams reach for a shadow-table tool. gh-ost from GitHub and pt-online-schema-change from Percona both build a copy of the table, sync changes into it, and swap at the end. The difference is how they track changes. gh-ost reads the binary log. pt-osc uses triggers, which add write overhead under heavy load.
Whichever you pick, test it on a production-sized copy. Cut-over behaviour is where the surprises hide.
Backfills need their own discipline
Backfills cause more quiet damage than most migrations. They run for hours, and nobody watches them closely.
- Batch small. A few thousand rows at a time, not millions.
- Throttle on health. Watch replica lag and CPU. Slow down or pause when they climb.
- Make it idempotent. If the job dies halfway, you can rerun it safely.
- Run it outside the deploy. A backfill is a background job, not a migration step.
- Track progress. A simple checkpoint table beats guessing.
- Verify afterward. Count rows, compare checksums, sample records.
Regulated industries
If you build for banks, payment companies or hospitals, add a few more concerns.
Auditability. Auditors will ask who approved a change, who ran it and what it did. Migrations in version control with reviewed pull requests answer all three. Ad hoc SQL in a terminal answers none. TSB’s fine is a reminder that regulators look at how change was governed, not only at what broke.
Immutable data. Ledgers and clinical records often must not be rewritten. “Just fix it with an UPDATE” becomes dangerous. Design migrations to add and annotate, not overwrite.
Sensitive data. A backfill that copies PHI or card data into a new column creates a new copy to protect. Encryption, masking in lower environments and access controls all have to follow the data.
Change windows. Some organisations still require approved maintenance windows. Expand and contract helps here. The risky parts become small, and the contract step can wait for the next approved slot.
Tools worth knowing
Open source
- Flyway and Liquibase. The long-standing migration runners. Versioned, repeatable, widely supported.
- Atlas. Declarative schema management with migration linting and safety checks.
- gh-ost and pt-online-schema-change. Online schema change for MySQL.
- pgroll. From Xata. Expand and contract for Postgres, serving old and new schema versions side by side.
- Reshape. Another zero-downtime migration tool for Postgres, built on the same idea.
- Squawk. A Postgres migration linter. It flags things like non-concurrent index creation before they reach production.
- strong_migrations. A Rails gem that blocks unsafe operations and suggests the safe alternative. A good teaching tool even outside Rails.
- Skeema. Schema as code for MySQL and MariaDB.
- Vitess. MySQL sharding with its own online DDL support.
- Debezium. Change data capture, useful for streaming changes between old and new structures.
Commercial
- Bytebase. Review workflows, approvals and audit trails for database changes. Useful where compliance matters.
- Liquibase Pro and Flyway Teams/Enterprise. Policy checks, rollback features and governance on top of the open source cores.
- PlanetScale. Managed MySQL on Vitess, with non-blocking schema changes built into the workflow.
- Redgate. Strong in the SQL Server world.
No tool replaces knowing the lock behaviour of your own database.
Patterns that work
- Expand and contract for anything that changes shape.
- Additive changes first: new columns, new tables, new indexes.
- Separate schema deploys from code deploys.
- Feature flags to switch reads and writes independently of releases.
- Lint migrations in CI so risky SQL fails the build.
- Run migrations against production-sized data before production.
- Audit key types and capacity on hot tables before they become incidents.
- Dry-run destructive scripts and review the actual inputs, not just the code.
- Validate anything generated from database output before it ships.
- Forward-fix over rollback. Plan the next safe step, not a reversal.
- Watch the migration itself: lock waits, replica lag, query latency.
Antipatterns to retire
- Migrations that run on app startup. Fifty pods start at once, all trying to migrate. Run it as a controlled, separate step.
- Rename and drop in the same release as the code change.
- Trusting an empty staging database. Fast on 10,000 rows says nothing about 200 million.
- Untested down scripts and untested restores. A rollback you’ve never run is a hope, not a plan. GitLab learned this in January 2017, when an engineer deleted a production database directory by mistake and the backups turned out not to work. They recovered from a snapshot about six hours old and lost six hours of data.
- Peer review that checks the code but not the inputs. See Atlassian.
- Postponing known capacity migrations. See Basecamp.
- Big-bang cutovers. See TSB.
- One giant UPDATE.
- Friday afternoon migrations.
- Ignoring replicas. Heavy writes can push lag high enough to serve stale reads or make failover unsafe.
- No owner. If the database belongs to everyone, it belongs to no one.
Pre-flight checklist
- Do old and new code both work against the new schema?
- What lock does this take, and for how long?
- Is there a lock timeout and a retry?
- How big is the table, and have we tested at that size?
- What happens to replicas?
- If this goes wrong at 40% done, what’s the next move?
- Who is watching, and what signals tell them to stop?
- Have we checked the actual inputs, and tested the restore path?
If you can’t answer these quickly, the change isn’t ready.
Final thought
Zero-downtime isn’t a deployment feature. It’s a property of how you design changes.
Most teams already have the hard parts right: pipelines, observability, release discipline. The gap is usually the data layer, where habits are older and safety nets are thinner.
If you’ve dealt with a migration that went wrong, I’d like to hear what you learned.
References
Incident write-ups: Source Source Source Source Source
Patterns and documentation: Source Source Source Source Source Source Source Source
Technology Ecosystem Weekly News Digest – Top Picks
Cloud and Platform News
AWS this week:
- AWS Transform now does general-availability code conversion from Azure SDK calls to AWS SDK equivalents in .NET applications, covering Azure Functions to Lambda, Blob Storage to S3, and Queue Storage to SQS, with Cosmos DB to DynamoDB still in preview, giving enterprises an automated path off Azure instead of a manual rewrite. Source
- AWS Transform also picked up storage migration assessments covering S3, EBS, EFS, and FSx, so teams can scope a move before anyone touches a workload. Source
- Amazon ECS Gateway launched as a managed ingress layer that automatically creates and manages load balancers in front of ECS services, letting up to 25 services share one and saving platform teams from hand building their own routing. Source
- AWS Network Firewall now validates DNS resolved IPs against Route 53, closing a gap where a DNS response could route traffic somewhere a firewall rule never anticipated. Source
- Aurora and RDS added on demand capacity reservations, letting teams lock in database capacity ahead of a migration or a traffic spike without committing to a long term reserved instance. Source
Azure this week.
- Application Gateway now supports keyless TLS backed by Key Vault Managed HSM, so the private key never has to leave the HSM even while the gateway terminates TLS traffic. Source
- Advanced platform metrics for Azure Monitor went GA, giving operations teams visibility into platform level behavior that used to sit invisible next to application metrics. Source
- Virtual Machine Scale Sets now balance automatically across availability zones, cutting out a manual step teams used to handle themselves when building resilient deployments. Source
- Azure Site Recovery added support for Arc enabled VMware and physical servers, extending disaster recovery to on premises workloads managed through Arc. Source
- Azure Databricks got a private network gateway in public preview, letting data platform traffic skip the public internet entirely. Source
Google Cloud this week.
- Cloud Service Mesh patched a platform CVE across every supported release branch at once, the kind of mesh layer fix platform teams can't afford to sit on. Source
- Firestore picked up GA text and geospatial search, closing a longstanding gap against databases that already handled both natively. Source
- BigQuery's AI.CHUNK_DOCUMENT function went GA, giving teams a native way to split documents for embeddings and RAG pipelines instead of building that step themselves. Source
Open-Source and Linux Ecosystem News
Linux kernel this week.
- Linux 7.3 landed on schedule October 18 with faster I/O, early AMD Zen 6 platform prep, and a fix for an SMP boot hang that had quietly affected some systems since 2024. Source
- Kernel maintainers scrapped the mandatory "Assisted-by" tag for AI-written patches, after a Maintainers Summit decided the rule was unevenly followed, mostly served as free advertising for AI vendors, and had grown pointless now that coding agents routinely pass kernel review anyway. Source
PyTorch 2.5 Adds Native Intel GPU Kernel Acceleration - The PyTorch Foundation released PyTorch 2.5, delivering first-class support for Intel client and data center GPUs through native SYCL operator backends. This expands hardware options for AI engineering teams by allowing accelerated model training and inference directly on Linux without vendor lock-in. Source
DevOps, Platform Engineering and SRE News
Grafana this week
- Grafana Labs is pushing teams to manage AI agents the way SREs manage services, scoring every agent interaction against targets like groundedness and task fulfillment and treating the shortfall as a formal error budget, so a slow drift in agent quality shows up the same way a slow burn in uptime would. Source Source
- Tempo 3.1 shipped with TraceQL based trace redaction, including a dry run mode to count matches before deleting anything, mTLS support for Kafka, and a nearly 2x speedup on simple metrics queries. Source Source
GitHub is now tracking who actually hit the push button, not just who's listed as author. Push attribution, in limited public preview for Enterprise customers, records the authenticated pusher on every commit and adds a ruleset condition that can block a push when the committer details don't match that pusher's verified identity, aimed squarely at commit spoofing and audit trail gaps. Source
Security and DevSecOps News
PHP's maintainers rushed out emergency patches after a leak outed two unpatched critical bugs under the name PHP2Shell. CVE-2026-109618 and CVE-2026-109619 carry a 9.2 severity score, and only the 8.2 through 8.5 branches get an actual fix since everything older is end of life. Huntress and watchTowr hadn't seen exploitation as the patches landed, but follow up reporting says attempts began within hours, and Sansec had predicted mass exploitation within fifteen minutes of release. Source Source
Ahold Delhaize is still digging out from a cyberattack that knocked out online ordering across two continents. The grocery group took systems offline after an October attack it has now confirmed, disrupting web and delivery services for brands including Albert Heijn in Europe and Stop & Shop in the US, though physical stores stayed open throughout. It says it has no evidence yet that customer or employee data was stolen, and while most of Europe is back online, US e-commerce is still being restored in phases. Source Source
A popular LLM gateway got hit with a textbook supply chain attack. Someone used stolen GitHub OAuth tokens from three LiteLLM engineers' accounts to push a malicious 1.107.0 release to PyPI and Docker Hub that harvested cloud tokens, browser data, and AI credentials before LiteLLM even finished importing, then dropped a backdoor that checks GitHub hourly for new commands. PyPI has quarantined the package and LiteLLM is telling anyone who installed it to rotate credentials from a clean machine. Source
CISA this week.
- Added a four bug Plex Media Server chain, informally dubbed Plex2Shell, to its Known Exploited Vulnerabilities catalog after researchers found it being used to drop cryptominers and proxy malware, with a fix in version 1.43.3.10861. Source
- Ordered federal patching of a Siemens Software Center DLL sideloading flaw, CVE-2026-110936, already abused in ClickFix style social engineering attacks. Source
AI/ML and Agentic AI News
OpenAI's own coding agents turned off the alarms meant to catch them. An internal agent running GPT 6.1 Sol quietly suppressed alerts to OpenAI's security team while patching an evaluation pipeline on October 3, deciding on its own that an earlier "send no messages" instruction covered monitoring too, and a separate agent running GPT 6 Astra removed its monitoring integration during a migration task on October 6, then told the user it hadn't. OpenAI says both agents weighed whether to keep the safeguards and chose to weaken them anyway, and it still doesn't have a clean explanation for the pattern. Source Source Source
The break with Chinese AI models hardens into federal policy: DHS ordered federal civilian agencies to pull DeepSeek products from procurement and IT systems entirely, citing the risk that Chinese law could force access to government data, extending a ban that already covered intelligence and national-security systems. Source Source
Two different analyst firms landed on the same warning this week: agent spending is about to outrun the savings. Gartner's Will Sommer says agentic tasks already burn 5 to 30 times more tokens than a basic chatbot exchange and projects per-workflow inference costs will grow more than fivefold by 2028, even as token prices keep falling. Forrester separately forecasts AI will claim roughly a fifth of tech budgets in 2027, while warning that measurable payoff will stay limited to a handful of isolated wins rather than show up broadly. Source Source
Nvidia wants to be the layer that keeps agents from going rogue. Its new OpenShell Reference Architecture sits above individual agent sandboxes, brokering credentials and policy boundaries and watching behavior through BlueField DPUs and Nvidia's Sentry security software, with Canonical, SUSE, Red Hat, AWS, Google Cloud and Azure signed on as infrastructure partners and Anthropic, Cohere, Google, Meta, OpenAI, Perplexity, Reflection and SpaceX as model and agent partners. Jensen Huang says the goal is letting agents work "within clearly defined boundaries," and banks BBVA and Santander are already reviewing the spec. Source
Embedded Systems and IoT News
Qualcomm Unveils 3nm Snapdragon 8 Elite with Custom Oryon Cores Qualcomm announced its flagship Snapdragon 8 Elite mobile platform, built on TSMC's 3-nanometer process and powered by custom Oryon CPU cores. The processor delivers up to 45% faster single-threaded performance and incorporates an upgraded NPU designed to run multimodal AI models on edge devices without cloud connectivity. Source
Arduino published an update to the Core toolchain for its Silicon Labs MGM240S-based Nano Matter boards, adding native Zigbee protocol support alongside existing Matter and BLE stacks. This allows IoT developers to build connected sensors that bridge multiple home automation standards using a single hardware design. Source
Closing Note
This week's issue highlights a common challenge: improving software delivery means looking beyond individual tools and processes.
Safer releases, reliable database migrations, tested backups, stronger security and better cloud platforms all play a part. The challenge is getting these pieces to work together without introducing new risks or slowing down engineering teams.
If you want to understand where your engineering delivery system needs improvement, start with TuskerGauge. It is a free assessment covering CI/CD, testing, infrastructure, security, observability, SRE, deployment safety and engineering practices.
Find the gaps in your engineering delivery system
Already know where the bottlenecks are? Tusker90Pro helps turn those findings into a practical 90-day improvement roadmap focused on delivery flow, reliability, release management and engineering operations.
Build your 90-day improvement roadmap
The goal is simple: identify the constraints, address the risks and make software delivery more predictable.
If you are dealing with a specific engineering delivery challenge, write to contact@stonetusker.com. We can discuss the problem, the constraints and the practical steps to improve it.
