Deploying Without Holding Your Breath

There’s a particular silence that follows pressing deploy. Everyone watches the dashboards. Nobody types anything. For about ninety seconds, an entire team is holding its breath.

That ritual exists because the normal Kubernetes deployment has exactly one gear: go. It replaces old pods with new ones as fast as it safely can, and its only question is “is the new pod alive?” — not “is the new pod any good?”

A pod can be perfectly alive and still be returning errors to every single customer.

The problem with “alive”

Kubernetes checks health the way a doorman checks ID. It knocks, the container answers, the container goes on the list. Whether that container is quietly 500-ing on every real request is not a question anyone asked.

So the real failure isn’t the bug — bugs are unavoidable. It’s that 100% of your users meet the bug at the same moment, and the clock on fixing it doesn’t start until a human notices.

The idea: a dial instead of a switch

What if, instead of flipping everyone over at once, you sent 5% of traffic to the new version and watched what happened?

If it looks fine, go to 20%. Then 50%. Then everyone. If it looks bad at any point, send everyone back to the old version — which is still running, untouched, ready.

Your blast radius stops being “everyone” and becomes “5% of people, for two minutes.”

That’s already a huge win. But there’s still a human in the loop, squinting at dashboards and deciding when to turn the dial. The interesting part is removing that human.

---

Three pieces, three jobs

Here’s where the three tools in the title come in, and the division of labour is cleaner than you’d expect.

Argo Rollouts replaces your Deployment with a Rollout — same thing, but it takes a recipe: go to 20%, wait, check, go to 40%…

Istio is the traffic valve. Kubernetes alone can only split traffic by changing pod counts (want 5%? run 19 old pods and 1 new one). Istio splits by percentage, independent of how many pods exist. That’s the difference between a dial and a staircase.

Prometheus is the judge. It already knows your error rate. The clever bit is that Istio reports every request it routes, so you get a per-version error rate for free — without adding a single line of instrumentation to your app.

That last point is the quiet hero of this setup. Istio sees v1 and v2 as separate destinations, so the question “is the new version worse than the old one?” becomes a query you can just ask.

---

The recipe

Here’s a complete canary strategy. Even without knowing the syntax, you can read it top to bottom:

strategy:
  canary:
    steps:
      - setWeight: 20          # send 20% to the new version
      - pause: {duration: 30s} # let it marinate
      - analysis:              # ask Prometheus: still healthy?
          templates:
            - templateName: success-rate
      - setWeight: 50
      - pause: {duration: 30s}
      - setWeight: 100         # all good — everyone moves over

And the judge it consults:

successCondition: result[0] >= 0.95
failureLimit: 3

In English: at least 95% of requests must succeed, and I’ll tolerate three bad readings before I pull the cord.

If the analysis fails, Argo doesn’t page anyone. It sets the traffic weight back to zero and stops. The old version — which never went anywhere — is serving 100% again, usually before a human would have finished reading the alert.

---

What this actually feels like

I’ve put the full local setup in a separate walkthrough (link below), but here’s the part worth seeing.

You deploy a deliberately broken version. Then you watch this:

Name:            rollouts-demo
Status:          ॥ Paused
Strategy:        Canary
  Step:          3/6
  SetWeight:     20
  ActualWeight:  20

NAME                        KIND         STATUS        AGE
⟳ rollouts-demo             Rollout      ॥ Paused      3m
├──# revision:2
│  └──⧉ rollouts-demo-7bf84f9696  ReplicaSet   ✔ Healthy     40s
│     └──□ ...                    Pod          ✔ Running     40s
└──# revision:1
   └──⧉ rollouts-demo-6cf78c9f55  ReplicaSet   ✔ Healthy     3m

Note what’s on screen: both versions, at once, with a weight between them. That’s the whole mental model in one view. Revision 1 hasn’t been destroyed; it’s sitting there at 80%, waiting to find out whether revision 2 earns its job.

Thirty seconds later the analysis runs, finds a 60% error rate, and the status flips to Degraded. Traffic goes back to 100% old version. Nobody did anything.

The first time you see that happen on your own machine, the idea stops being abstract.

---

Three honest caveats

Canary needs traffic to work. The judge is doing statistics. If your service gets ten requests an hour, 5% of that is nothing to measure, and the analysis is theatre. For low-traffic services, blue-green (keep both versions, switch all at once, keep the old one warm) is the more honest choice.

A good metric is harder than the setup. Error rate catches crashes and 500s. It does not catch “the checkout button now charges twice” — that’s a success, by HTTP’s standards. Your analysis is exactly as smart as the question you ask it.

Istio is a big thing to adopt. A service mesh is real operational weight. If you don’t already run one, Argo Rollouts can split traffic by pod count without it — coarser, but zero new infrastructure. Don’t adopt a mesh just for canaries.

Try it locally

The full walkthrough — local cluster, Istio, Prometheus, the manifests, and a deliberately broken version to watch get rejected — is here:

→ Argo Rollouts local demo: canary with Istio + Prometheus

It’s about twenty minutes end to end, and everything runs on your laptop. The reference demo video is Argo Rollouts Demo, which covers the same Istio integration.

The short version

• Kubernetes checks if a pod is alive, not whether it works. Those aren’t the same claim.
• A canary makes the blast radius small: 5% of users for two minutes, instead of everyone at once.
• Three jobs: Argo Rollouts conducts, Istio splits the traffic, Prometheus judges the result.
• Istio gives you per-version error rates free, because it already sees every request and knows which version answered it.
• The rollback is the point. The old version never stopped running, so going back is a weight change, not a redeploy.
• Automated analysis removes the human from the loop — and the human was always the slowest part.

The goal isn’t deploying more often. It’s that deploying stops being an event you need to be present for.