Your Kubernetes Cluster Is Not as Highly Available as You Think

We recently spent a week hardening a production platform: an ingress layer, an identity provider, and about 150 microservices across three clusters. Everything was "already HA". Two or three replicas of everything, spread across availability zones, health checks in place.

Then we looked closer. Here is what we found, what we changed, and why every one of these matters more than the last one.

1. The PodDisruptionBudget that protected nothing

A PodDisruptionBudget (PDB) tells Kubernetes how many pods of an app may be down at once during voluntary disruptions: node drains, cluster upgrades, autoscaler consolidation. Without one, a drain happily evicts both replicas of your payment service at the same time.

The platform had PDBs on 130 services. Good. Every one of them selected pods by a label called app. The pods carried app.kubernetes.io/name instead. Zero pods matched. For months, every PDB had been reporting expectedPods: 0 and nobody had noticed, because a PDB that matches nothing never blocks anything and never complains.

Lesson: a PDB is only as good as its selector. Check kubectl get pdb and look at the ALLOWED DISRUPTIONS column. If it says 0 on a healthy app with several replicas, something is wrong.

2. Topology spread: "soft" means "optional"

Pods were told to spread across zones and nodes with whenUnsatisfiable: ScheduleAnyway. That is a preference, not a rule. The scheduler honours it when convenient.

On one cluster we found the ingress controller running three replicas: two on the same node, and an entire availability zone with none. Karpenter had consolidated nodes overnight and the soft rule quietly gave way.

We made the zone constraint hard (DoNotSchedule) and left the node constraint soft. Hard on zones means an AZ outage can never take out every replica. Soft on nodes means pods still pack efficiently instead of demanding a node each.

Two details most people skip:

• matchLabelKeys: [pod-template-hash] makes the spread rule count only pods of the current rollout. Without it, old pods still draining prop up the count, and a rolling update can finish with two new pods in one zone and none in another. We worked through the arithmetic; it happens more easily than you would think.
• nodeTaintsPolicy: Honor stops nodes your pod can never run on, like a tainted system node group, from counting as an empty zone the scheduler feels obliged to fill.

3. Requests and limits: the memory rule

Set memory request equal to memory limit. Always. A pod whose request is lower than its limit is a promise the node may not be able to keep, and the kernel resolves that by OOM-killing someone, often not the culprit.

Leave CPU without a limit. CPU throttling makes latency worse without protecting anyone. Requests alone are enough for scheduling.

We size these from real usage with KRR rather than guessing, then revisit quarterly.

4. Probes: the difference between "up" and "ready"

Three probes, three jobs:

• Startup gives a slow-booting app time to load before anyone judges it. A Java identity provider needed a minute; without a startup probe, liveness would have killed it in 30 seconds and looped forever.
• Readiness decides whether traffic should reach the pod. This is what removes a pod from the load balancer during a rollout or shutdown.
• Liveness restarts a pod that is genuinely stuck.

The ingress controller shipped with a readiness probe that failed after one missed check with a two second timeout. One slow garbage collection pause and a perfectly healthy pod dropped out of rotation. We loosened it to three failures and five seconds. Tolerant readiness, strict liveness.

5. Rollout strategy: never go below capacity

maxUnavailable: 0, maxSurge: 1. Start a new pod, wait until it is ready, then stop an old one. Capacity never dips. This does not always apply to statefulsets or operator managed workloads so always refer to the application's documentation.

6. Graceful termination: the five seconds nobody adds

When Kubernetes stops a pod, two things happen at the same instant: the app receives SIGTERM, and the pod is removed from the load balancer's endpoint list. The second one takes a couple of seconds to propagate. In that gap, live traffic hits a pod that is already shutting down, and users see 502s.

The fix is a preStop hook that sleeps for five seconds before SIGTERM. The app keeps serving while the network catches up. We then set terminationGracePeriodSeconds high enough to cover the sleep plus the app's own drain time.

We tested it the honest way: a request loop through the public endpoint at ten requests per second while deleting pods behind it. Hundreds of requests, zero failures, pods gone in eight seconds.

The part that surprised us

None of these settings are exotic. Most were "already configured". The gap was between configured and working.

High availability is not a checklist you tick once. It is something you verify, on the real cluster, by breaking things on purpose and watching what happens.

That is what we do. If you would like us to do it to your platform before an incident does it for you, get in touch.