---
title: "Hardening Kubernetes for Production: PDBs, Autoscaling, and Admission Gates"
description: "Field-tested operational patterns for running Kubernetes in production: Pod Disruption Budgets, resource limit admission controls, and resilient health probe design."
canonical: "https://zainahmed.net/blog/kubernetes-production"
published: "2024-01-08"
---

# Hardening Kubernetes for Production: PDBs, Autoscaling, and Admission Gates

> Author: Zain Ahmed
> Published: 2024-01-08
> Read Time: 2 min read
> Category: Kubernetes
> Tags: Kubernetes, DevOps, Production, SRE
> Canonical URL: https://zainahmed.net/blog/kubernetes-production

## Excerpt
Field-tested operational patterns for running Kubernetes in production: Pod Disruption Budgets, resource limit admission controls, and resilient health probe design.

---

Moving workloads from local staging clusters to production Kubernetes environments requires engineering around failure as a standard operational state. Node drains during cloud provider maintenance, bursty traffic spikes, and memory leaks will bring services down unless explicit resilience primitives are configured. Here are the essential safeguards required for production-ready Kubernetes deployments.

## 1. Pod Disruption Budgets (PDBs)

Cloud upgrades, Karpenter node rightsizing, and security patching drain nodes continuously. Without a Pod Disruption Budget, cluster autoscalers can evict every pod of a single deployment simultaneously, triggering an unexpected outage.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-service
```

- **`minAvailable` vs. `maxUnavailable`:** Declare `minAvailable: 2` (or `minAvailable: 80%`) to ensure the Kubernetes eviction API halts node drains until healthy replacement pods are running on alternative nodes.
- **Multi-Replica Invariant:** Never set `minAvailable: 1` on a deployment with `replicas: 1`; doing so locks node drains indefinitely and prevents cluster upgrades.

## 2. Resource Requests, Limits, and CPU Throttling

Omitting resource requests leads to catastrophic "noisy neighbor" scenarios where a runaway batch job exhausts node memory and causes Linux kernel OOM kills on critical API pods.

- **Set Equal Memory Requests and Limits:** Setting memory requests equal to limits (`requests.memory: 1Gi`, `limits.memory: 1Gi`) places pods in the `Guaranteed` Quality of Service (QoS) class, protecting them from eviction during node pressure.
- **Beware CFS CPU Quota Throttling:** Setting aggressive CPU limits can throttle application threads even when the host node has surplus CPU capacity. Monitor `container_cpu_cfs_throttled_periods_total` in Prometheus. Prefer sizing CPU requests accurately based on measured p95 utilization while omitting or raising CPU limits to prevent artificial latency spikes.
- **Kyverno / OPA Admission Enforcement:** Deploy an admission controller (such as Kyverno) to reject any Deployment or StatefulSet manifest that lacks explicit CPU and memory request definitions.

## 3. Designing Resilient Health Probes

Misconfigured probes are one of the most common causes of cascading cluster failures:

- **Decouple Liveness from Downstream Databases:** A liveness probe should only test whether the internal container process is responsive (e.g., checking an in-memory HTTP `/healthz` endpoint). If a liveness probe queries an external PostgreSQL database or Redis cache, a momentary database outage causes Kubernetes to restart every application pod at the same time, overwhelming the database upon recovery.
- **Use Readiness Probes for Traffic Shedding:** Readiness probes (`/ready`) should check downstream connectivity and worker thread availability. When a downstream dependency degrades, the pod is removed from the Service endpoints without being killed, allowing in-flight requests to complete gracefully.
- **Configure `failureThreshold` and `periodSeconds`:** Set `failureThreshold: 3` and `periodSeconds: 10` to avoid restarting pods during transient 1-second network blips.

---

## Related Endpoints & Navigation
- **All Articles**: [https://zainahmed.net/blog](https://zainahmed.net/blog)
- **Author Profile**: [https://zainahmed.net/about](https://zainahmed.net/about)
- **Machine-Readable LLM Manifest**: [https://zainahmed.net/llms.txt](https://zainahmed.net/llms.txt)
- **RSS Feed**: [https://zainahmed.net/feed.xml](https://zainahmed.net/feed.xml)