Release Trains for AI-Assisted Products

Weekly model change is not a release process

AI-assisted products accumulate change from three directions at once: product behavior, model provider defaults, and the evaluation suites that decide whether quality is acceptable. Teams that treat each of those as an ad-hoc hotfix discover that 'shipping AI' becomes a permanent interrupt queue for engineers and a permanent surprise for support.

A release train is the antidote. It does not slow learning; it batches change behind a predictable window so product, platform, trust, and support can prepare. The train leaves on schedule whether or not every ticket is ready. Incomplete work waits for the next departure instead of forcing an unplanned deploy.

Version the three surfaces together

Prompts, retrieval configuration, tool schemas, and evaluation fixtures are one system. Shipping a prompt without the suite that proves its properties is not iteration; it is unmeasured risk. Likewise, bumping a model version without pinning expected cost, latency, and refusal behavior turns provider changelogs into production incidents.

Keep a single release artifact that names the model route, prompt package, retrieval index revision, and eval suite hash. That artifact is what enters canary, what rolls back, and what gets discussed in postmortems. If you cannot reconstruct those four values for a bad answer, you do not have a release process.

• Pin model IDs and temperature; never ship on 'latest'
• Co-version prompts with golden cases and graders
• Record retrieval index and tool schema revisions on every train
• Reject merges that change AI behavior without an eval delta

Gates must reflect product risk, not demo polish

A train that only checks that the model still answers is theater. Gate on the properties users and the business cannot lose: groundedness for knowledge features, schema validity for structured outputs, policy refusals for high-risk intents, cost per successful outcome, and p95 latency under realistic load.

Risk-tier the features on the train. Drafting aids can tolerate softer bars and broader canaries. Actions that write to customer data, spend money, or send external messages need stricter evals, smaller exposure steps, and an explicit human-owned go/no-go. The calendar is shared; the bar is not.

Coordinate humans around the cadence

Release trains fail when they are an engineering ritual with no product counterpart. Publish the departure schedule, freeze windows for high-stakes features, and give support a short brief of what changes for users. AI regressions often look like 'the product got worse' rather than 'the deploy failed', so communication is part of reliability.

Use the train to absorb provider-driven change as well. When a model vendor updates defaults, schedule the adaptation on the next train instead of hot-patching production at midnight. Predictability is the staff-level outcome: fewer emergency deploys, clearer ownership, and a product that can still move weekly without gambling the brand on every prompt edit.

• Google SRE — Continuous Delivery
• LaunchDarkly — Progressive delivery overview
• OpenAI — Production best practices