Deploys a new version to a small percentage of production traffic first, monitors error budgets and latency against baseline, and automatically widens or rolls back based on pre-defined criteria. This keeps the blast radius of a bad deployment small—particularly important when AI agents are modifying deployment pipelines where a single bad command could affect many users.
Use cases
- Rolling out a risky dependency upgrade where you want early signals before committing to full deployment
- Deploying a new AI model version or prompt change that could affect response quality in subtle ways
- Friday deployments where you want to limit exposure over the weekend when fewer engineers are available
- A feature flag toggle for a high-traffic feature where you want to validate performance before the full audience
- Deploying infrastructure changes (new database version, new caching layer) where behavior differences are not obvious in staging
Key features
- Before shifting any traffic, define success metrics: error rate, latency p99, and any model-quality metrics appropriate to the change
- Set the initial canary slice to a small, representative subset of traffic—typically 1-5% of requests—and route it to the new version
- Monitor the success metrics continuously for the first 30-60 minutes and compare against the baseline from the previous stable version
- If metrics stay within acceptable bounds, automatically widen to 25%, then 50%, then 100% on a pre-defined schedule; if metrics degrade, automatically roll back to the previous version
- After full rollout, confirm metrics remain stable for at least one full business day before considering the deployment complete
When to Use This Skill
- When deploying any change that could affect a significant percentage of users and rollbacks are costly
- When rolling out AI model or prompt changes where quality degradation is hard to detect in staging
- When deploying on high-stress days (Monday morning, Friday afternoon, holiday periods) when incident response is slower
Expected Output
A canary deployment with defined success metrics, automated promotion/rollback rules, and a deployment record documenting the rollout progression from canary to full.
Frequently Asked Questions
- How do I choose the right canary percentage?
- Start with 1-5% and widen in stages. The right percentage depends on how much traffic you need to detect a meaningful signal. For high-traffic services, 1% may be enough to detect a 1% error rate increase. For low-traffic services, you may need 20% or more.
- What if automated rollback triggers on a false positive?
- Review the metrics that triggered the rollback: if the baseline was already degraded before the deployment, the rollback was unnecessary and you should adjust the rollback threshold. If the signal was noise, increase the confirmation window before triggering rollback.
- How does canary deployment work with GitOps and infrastructure-as-code?
- In GitOps, the desired state is declared in git. A canary is implemented by temporarily routing a percentage of traffic to a different version of the declared state, while the git state remains the source of truth for the full rollout.
Related
Related
3 Indexed items
Observability baselines
Establishes golden signals (latency, traffic, errors, saturation), SLO windows, and dashboard checks before agents automate deployments so that 'healthy' and 'degraded' have measurable definitions rather than subjective interpretations. This is essential when AI agents are managing deploys because agents need objective metrics to make decisions, not human gut feelings.
Postmortem writing
Captures the full incident timeline, blast radius, contributing factors, and concrete follow-up actions after production incidents so teams build institutional memory rather than repeating the same surprises. A well-written postmortem separates root cause from triggers, avoids blame, and produces tracked action items that prevent recurrence.
Agentic coding vendor readiness review
Turns platform reliability and multi-vendor coding-agent guidance into a checklist before standardizing on a single AI coding stack. Teams inventory host-platform SLAs (for example GitHub availability incidents documented on githubstatus.com), compare primary and backup agents (GitHub Copilot, Cursor, Claude Code, Codex, etc.), verify observability hooks through Braintrust or similar tracing, and rehearse workflows when the code host or agent API is degraded. The skill cites public status pages and vendor billing changes—such as usage-based Copilot pricing announced on github.blog—so procurement and engineering sign off with eyes open about downtime, leadership churn, and feature parity gaps reported in trade media.