In 2015, most engineering teams ran containers by hand, one `docker run` at a time. A decade later, Kubernetes alone processes an estimated millions of pods across the world’s cloud infrastructure, and the OCI’s Open Container Initiative standards underpin nearly every production deployment. Yet the gap between “it runs on my laptop” and “it survives Black Friday” is still where most teams stumble. The practices below are the ones that actually hold up when traffic spikes, nodes die, and auditors show up. (Kubernetes)
Start With Declarative Manifests, Not Imperative Commands
The single most common mistake is treating orchestration like a shell session. Engineers patch live objects with `kubectl edit`, tweak replicas by hand, and then wonder why staging and production drift apart within a week. Declarative configuration — everything in version control, applied through a pipeline — is the foundation everything else rests on.
In practice, this means:
- Store every manifest in Git, including ConfigMaps and secrets references (never raw secrets).
- Use a GitOps controller or CI pipeline as the only path to the cluster.
- Review manifest changes like code, with at least one approval.
- Tag releases so you can roll back to a known-good state in seconds.
Teams that skip step three usually discover the cost during an incident, when nobody remembers who changed the resource limits.
Set Resource Requests and Limits Deliberately
Kubernetes schedules pods based on resource requests and enforces ceilings with limits. Get this wrong and you get two failure modes at once: nodes that look full but run idle, and pods that get evicted the moment memory pressure hits. (Kubernetes Overview)
| Configuration | QoS Class | Eviction Risk | Best For |
|---|---|---|---|
| Requests = limits (CPU and memory) | Guaranteed | Lowest | Databases, stateful services |
| Requests set, limits higher or absent | Burstable | Medium | Most stateless web services |
| No requests or limits | BestEffort | Highest | Batch jobs you can lose |
The practical rule: always set memory requests equal to memory limits for anything stateful, because memory is not compressible. CPU is different — a container that hits its CPU limit gets throttled, not killed, so slightly generous CPU limits are usually fine. Measure real usage for two weeks before picking numbers; guessing produces either waste or outages.
Wire Up All Three Probe Types
Health checks are where well-meaning teams accidentally create cascading failures. A liveness probe that fires too early restarts a container mid-startup, which restarts it again — a loop that looks like an application bug but is a configuration bug.
- Startup probes give slow-booting apps time to initialize before other checks begin.
- Readiness probes control traffic. A failing readiness probe removes the pod from the Service endpoints without killing it.
- Liveness probes restart genuinely stuck processes. Use them sparingly and with generous thresholds.
A blunt but effective heuristic: if you can only add one probe, add readiness. It prevents the most user-visible damage. Point probes at a lightweight endpoint that checks dependencies, not a full-page render.
Plan for Disruption Before It Happens
Nodes get drained. Clusters get upgraded. Cloud providers reclaim spot instances with two minutes’ notice. Pod Disruption Budgets (PDBs) are the documented mechanism for capping voluntary disruptions so a routine maintenance window doesn’t take your whole service offline.
Pair PDBs with these habits:
- Spread replicas across availability zones with topology spread constraints or anti-affinity rules.
- Run at least three replicas for anything customer-facing.
- Test node drains in staging on a schedule, and repeat those tests outside incident response.
- Set `terminationGracePeriodSeconds` to match how long your app actually needs to finish in-flight requests.
The teams that handle disruptions well treat chaos testing as routine hygiene. The teams that don’t usually learn about missing PDBs at 3 a.m.
Scale on the Right Signal
The Horizontal Pod Autoscaler is the built-in answer for scaling workloads, but CPU-based scaling is often the wrong lever. A queue worker that’s I/O-bound will sit at 15% CPU while its backlog grows to six figures. Custom metrics — queue depth, request latency, active connections — tell a far more honest story.
Two practical cautions. First, autoscaling without resource requests is meaningless, because the scheduler has no basis for comparison. Second, scale-up is fast but scale-down is deliberately slow; configure stabilization windows so you don’t thrash between replica counts during bursty traffic.
Treat Security as Configuration, Not an Afterthought
Container security failures are rarely exotic exploits. They’re usually a container running as root with a writable filesystem and a service account token mounted by default. The fixes are configuration-level and cheap:
- Run as non-root with a read-only root filesystem where possible.
- Drop Linux capabilities and apply seccomp profiles.
- Use namespaces and network policies to segment workloads.
- Scan images in CI and block critical vulnerabilities from reaching the registry.
- Rotate credentials and avoid long-lived cluster-admin tokens.
This is also where AI governance discipline pays off: if your platform team uses AI-assisted tooling to generate manifests or triage alerts, those systems need the same review gates and audit trails as human changes. The principles behind AI governance, risks, and best practices map cleanly onto cluster operations — least privilege, traceability, and human sign-off on consequential actions.
Standardize the Platform Layer
Once you’re running more than a handful of services, the bottleneck stops being Kubernetes and starts being the developer experience. Every team hand-rolling its own Helm chart is a team that will drift.
What works in practice: a small set of golden templates that encode your organization’s defaults — probes, resource requests, PDBs, labels, logging sidecars — so teams inherit good behavior instead of remembering to add it. A service catalog or internal developer portal helps, but a well-maintained chart repository gets you 80% of the benefit for 20% of the effort.
This is also where orchestration tooling starts to overlap with coordination. As platforms grow, teams increasingly need to decide which system owns which decision — the scheduler, the autoscaler, the CI pipeline, or the human on call. Understanding what AI orchestration means in this context is useful framing: the same questions of ownership, handoff, and failure isolation apply whether the actors are containers or models.
Observe Both the Cluster and the App
Application monitoring tells you the service is slow. Cluster monitoring tells you why — a node with a saturated disk, a kubelet that can’t reach the API server, a pod stuck in `Pending` because no node fits its resource request.
Track these at minimum:
| Signal | Why It Matters |
|---|---|
| Pod restart counts | Early warning for probe misconfiguration and OOM kills |
| Pending pods | Capacity or scheduling constraint |
| Node resource pressure | Eviction risk before it happens |
| API server latency | Cluster-wide degradation indicator |
| Certificate and token expiry | Silent failure that hits at the worst time |
Set alerts on trends as well as thresholds. A slow climb in restart counts over three days is a far better signal than a single spike.
Keep Images Small and Runtimes Current
Large images slow every deploy, and slow deploys make teams afraid to ship. Multi-stage builds, slim base images, and a proper `.dockerignore` routinely cut image size by 70% or more. Pin base image digests rather than floating tags so a compromised or broken upstream tag can’t silently change your production artifact.
On the runtime side, the ecosystem moved on from the built-in Docker integration years ago; containerd and CRI-O are the standard container runtimes in modern clusters. Keep them patched, and treat cluster upgrades as a scheduled activity with a rollback plan rather than an emergency response.
Conclusion
Container orchestration rewards boring discipline. Declarative manifests, honest resource numbers, all three probe types, disruption budgets, signal-appropriate autoscaling, locked-down containers, and a platform layer that makes the right thing the easy thing — none of these are glamorous, and all of them are what separates clusters that hum along from clusters that page you every weekend. Start with the two changes that cost the least and prevent the most pain: put every manifest in Git, and set memory requests equal to memory limits on anything stateful. The rest can follow.
Frequently Asked Questions
What is the most important container orchestration best practice?
Declarative, version-controlled configuration. Every other practice — resource tuning, probes, disruption budgets, security policy — depends on having a single source of truth you can review, diff, and roll back. Teams that manage clusters imperatively lose the ability to reproduce or audit their own state.
How many replicas should a production service run?
At least three for anything customer-facing, spread across availability zones. Two replicas means a single node drain can take you to zero capacity during a rolling update. Three gives you one replica of headroom while still tolerating a zone failure.
Should I always set CPU limits?
Not necessarily. CPU is a compressible resource, so hitting a limit causes throttling rather than termination. Many teams set CPU requests carefully but leave CPU limits off, or set them generously, to avoid latency spikes from throttling. Memory is the opposite: always set memory limits, ideally equal to requests.
What is the difference between a liveness and readiness probe?
A readiness probe controls whether the pod receives traffic — failing it removes the pod from Service endpoints without restarting anything. A liveness probe controls whether the container keeps running — failing it triggers a restart. Confusing the two is one of the most common causes of self-inflicted outages.
How do I prevent maintenance from taking down my service?
Use Pod Disruption Budgets to cap how many replicas can go offline at once, combined with topology spread constraints so replicas don’t share a failure domain. Then rehearse: drain a node in staging regularly so the first time you test the process isn’t during a real upgrade.
Do I need a service mesh to run containers in production?
No. Most teams get a long way with built-in primitives: Services, Ingress, network policies, and solid observability. A service mesh adds real value for mTLS everywhere, fine-grained traffic shifting, and deep per-request telemetry — but it also adds operational surface area. Adopt one when you have a specific problem it solves, not because it’s on a maturity model diagram.
What belongs in a golden template versus a shared platform service?
Golden templates should carry the defaults every service needs and no team should have to remember: probe definitions with sane thresholds, resource requests and limits, PDBs, standard labels, logging and metrics sidecars, and security context settings. Shared platform services — ingress, certificate management, secret storage, observability backends — belong behind an API the template consumes, not copied into it. The dividing line is ownership: if a change requires coordinating across every team’s repository, it’s a platform service; if it’s a per-service default, it belongs in the template.
How do I know when to move off a single cluster?
Move when a constraint becomes structural rather than incidental: noisy-neighbor problems that resource quotas can’t contain, compliance boundaries that demand physical separation, or upgrade cadence conflicts between teams. A single well-run cluster with namespaces, quotas, and network policies handles far more than most teams assume. Splitting too early multiplies operational cost — every cluster needs its own upgrade path, certificate rotation, and monitoring — without solving the underlying problem.
What should we measure before tuning autoscaling?
Baseline your workload for at least two weeks: peak and median CPU and memory per replica, request latency percentiles, queue depth where applicable, and how quickly traffic actually shifts. Then choose the signal that moves first when load arrives. If CPU stays flat while latency climbs, CPU-based scaling will never fire in time. Set a stabilization window long enough to cover your traffic’s natural burst cycle, and verify scale-down behavior in staging before trusting it in production.
