Quick Answer: What are the most common mistakes when using kubernetes in production?
The most frequent Kubernetes production mistakes are missing resource requests and limits, absent PodDisruptionBudgets, deployments whose replicas all land on one node, and probes that are misconfigured or missing entirely. Most of these failures trace back to treating a cluster like a development sandbox long after it started serving real traffic, and they usually surface during upgrades, node drains, or traffic spikes rather than on quiet days.
Comparable but different problems include over-permissive RBAC, no network policies, and skipping etcd backups. Those are security and recovery gaps rather than availability gaps, and they often stay invisible until an incident forces a restore or an audit.
Before acting, check resource requests and limits, disruption budgets, topology spread and anti-affinity rules, probe configuration, version skew policy, RBAC scope, network policy coverage, image tagging discipline, and your etcd backup and restore drill history. (Configuration Best Practices)
A widely shared industry figure puts the scale of the problem in perspective: most organizations running Kubernetes in production have hit at least one outage caused by configuration, not code. The cluster rarely fails because the application is bad. It fails because the platform around the application was never finished.
That distinction matters. Kubernetes is a control plane, not an operations team. It schedules, reconciles, and restarts, but it does not decide that three replicas on one node is a bad idea, that a liveness probe checking a database connection will cause a restart storm, or that nobody has tested an etcd restore since the cluster was built. Those decisions belong to people, and the mistakes below are the ones people keep making.
Mistake 1: Shipping Pods Without Resource Requests and Limits
This is the mistake that quietly causes the most damage. A pod with no CPU or memory request is scheduled almost arbitrarily, because the scheduler has no signal about what the workload needs. A pod with no limit can consume a whole node’s memory and trigger evictions that take down unrelated services.
In practice, the fix is unglamorous: set requests based on observed usage, set memory limits close to requests, and be more generous with CPU limits than memory limits because CPU is compressible and memory is not. Teams that skip this step usually discover it during a traffic spike, when the noisiest service starves everything else on the node.
Mistake 2: Forgetting PodDisruptionBudgets
Voluntary disruptions are the ones you cause yourself: node drains, cluster upgrades, autoscaler scale-downs. Without a PodDisruptionBudget, Kubernetes is free to evict every replica of a service at once, and a routine maintenance window becomes an outage.
The pattern that works is a PDB with `minAvailable` set to a majority of replicas, paired with enough replicas that the budget is meaningful. A PDB on a single-replica deployment does nothing except block drains forever, which is its own kind of failure. If you are planning a control plane refresh, the drain behavior described in this walkthrough of an EKS cluster upgrade to Kubernetes 1.37 in 13 steps is exactly where missing PDBs turn into customer-visible downtime. (Kubernetes)
Mistake 3: Letting All Replicas Land on One Node
High availability is a scheduling property, not a deployment property. Three replicas of a service sound resilient until all three sit on the same node, and then a single hardware failure or node reboot takes the service to zero.
Topology spread constraints and pod anti-affinity are the documented remedies. The common trap is writing anti-affinity rules so strict that pods stay Pending during a scale-up, so the practical approach is preferred (soft) anti-affinity for most workloads and required (hard) rules only for the genuinely critical ones.
Mistake 4: Misunderstanding Liveness, Readiness, and Startup Probes
Probes are where good intentions cause restart loops. A liveness probe that checks a downstream dependency will restart a perfectly healthy pod every time the database hiccups, turning a small blip into a cascading failure. A missing readiness probe sends traffic to pods that are still warming up.
The distinction worth memorizing: liveness asks “should this container be restarted,” readiness asks “should this container receive traffic,” and startup asks “has this container finished booting.” Slow-starting applications should use a startup probe rather than an inflated liveness `initialDelaySeconds`, which delays every restart forever.
| Probe | Question it answers | Common mistake |
|---|---|---|
| Startup | Has the app finished booting? | Replaced with a long initial delay |
| Readiness | Should traffic reach this pod? | Missing entirely |
| Liveness | Should this container restart? | Checks external dependencies |
Mistake 5: Never Testing an etcd Restore
etcd is the cluster’s source of truth. Backups that have never been restored are not backups, they are hope. Teams routinely enable snapshots and then discover during an incident that the snapshot bucket was misconfigured, the encryption key was rotated away, or the restore procedure takes four hours nobody has.
The discipline that actually works is a scheduled restore drill into a scratch cluster, run on a calendar, with the timing recorded. If the restore takes longer than your recovery objective allows, you have found the problem before a real outage did.
Mistake 6: Ignoring Version Skew and Upgrade Order
Kubernetes supports only a limited version skew between the API server, kubelet, and kubectl. Upgrading nodes before the control plane, or jumping multiple minor versions at once, produces failures that look mysterious and are actually documented policy violations.
The reliable sequence is control plane first, then kubelets, then tooling, one minor version at a time, with the API deprecations for the target version reviewed in advance. Removed APIs are the silent killer here: a manifest that applied cleanly for two years can fail outright after an upgrade because the resource version was retired.
Mistake 7: Over-Permissive RBAC and Shared Credentials
Binding `cluster-admin` to a service account because it was the fastest way to unblock a CI pipeline is one of the most frequently cited findings in Kubernetes security guidance, including the NSA/CISA hardening guidance published in 2021 and revised in 2022. The same pattern shows up with long-lived kubeconfig files copied between laptops.
Least privilege in Kubernetes is tedious but tractable: namespace-scoped Roles instead of ClusterRoles, one service account per workload, and short-lived tokens issued by the platform rather than static credentials. Teams that have done it consistently report the audit burden drops sharply afterward.
Mistake 8: No Network Policies
By default, every pod can talk to every other pod. In a flat cluster, a compromised frontend can reach the database directly. NetworkPolicy is the control that fixes this, but it only works if the CNI plugin enforces it, and plenty of clusters run CNIs that silently ignore policy objects.
The practical check is a deliberate test: apply a deny-all policy in a staging namespace and confirm traffic actually stops. If it does not, the policy layer is decorative.
Mistake 9: Unbounded Image Tags and Sloppy Pull Policies
Deploying `:latest` means two pods in the same ReplicaSet can run different code, rollbacks have no defined target, and the audit trail disappears. Combined with a default image pull policy, it also produces surprise rollouts when a node restarts and re-pulls whatever the tag points to now.
Immutable tags, ideally digests, plus an explicit pull policy, plus a registry that enforces immutability, is the combination that holds up. This is the same reproducibility principle that makes automated asset pipelines trustworthy, whether you are pinning container images or generating consistent visuals with an AI mockup generator for product design.
Mistake 10: Treating kubectl apply as a Deployment Pipeline
Running `kubectl apply` from a laptop or a shell script works until it does not. There is no drift detection, no approval trail, no rollback story, and no record of who changed what at 2 a.m. GitOps tooling and proper CI pipelines exist precisely because this pattern fails at scale.
The related trap is assuming automation removes toil. Research on how AI assistants reshape engineering work found that automation frequently generates new review and verification work rather than eliminating it, a pattern described in this analysis of an Anthropic study on AI creating more work. Platform tooling follows the same rule: every layer you add needs an owner.
Mistakes That Only Appear Under Real Traffic
Some failures are invisible in staging because staging never sees the load. Connection pool exhaustion, insufficient horizontal pod autoscaler warm-up, and missing readiness gates all behave differently at production request rates.
The teams that catch these early share a habit: they load-test against a production-like cluster with production-like limits, and they watch the metrics that matter, including pending pods, eviction counts, and probe failure rates. It also helps to keep operational friction low for the humans on call. Something as small as knowing the fastest way to take a screenshot on a Mac during an incident review, or how to take a screenshot on Windows when documenting a postmortem, sounds trivial until you are the person writing the timeline at 3 a.m.
How to Audit Your Cluster This Week
A short, ordered checklist beats a grand remediation plan:
- List every deployment without CPU and memory requests.
- List every deployment without a PodDisruptionBudget.
- Check whether any service has all replicas on one node.
- Review probes for external dependency checks in liveness.
- Confirm the last successful etcd restore, with a date.
- Audit every `cluster-admin` binding and its justification.
- Verify network policies are actually enforced by the CNI.
- Grep manifests for `:latest` and floating tags.
- Confirm no workload is deployed by hand from a workstation.
Most clusters fail at least three of these on the first pass. That is normal, and it is also the point: the mistakes are not exotic, they are simply unglamorous enough to keep getting deferred.
Conclusion
Kubernetes in production fails in predictable ways. Missing resource controls, absent disruption budgets, collapsed topology, misunderstood probes, untested backups, ignored version skew, loose RBAC, decorative network policies, floating image tags, and manual deploys account for the overwhelming majority of incidents. None of them require deep expertise to fix. They require someone to decide that the platform deserves the same rigor as the application running on it. Start with the audit list, fix the three worst gaps, and schedule the next pass before the cluster schedules it for you.
Frequently Asked Questions
Why do resource limits cause more outages than they prevent?
Memory limits that are set too low trigger OOMKills under normal load spikes, and CPU limits that are too tight cause throttling that looks like application slowness. Limits protect the node, but they must be derived from observed usage rather than copied from a tutorial. Start generous, measure, then tighten.
Is a single-replica deployment ever acceptable in production?
For stateless critical services, no. For internal tooling with an accepted downtime window, sometimes. The decision should be explicit and documented rather than accidental, because a single replica combined with a node drain is a guaranteed outage.
How often should etcd backups be restored in a test environment?
A common cadence is quarterly for smaller platforms and monthly for clusters with tight recovery objectives. The restore should be timed, because the number that matters is minutes to recovery, not whether the snapshot file exists.
What is the difference between a PodDisruptionBudget and pod anti-affinity?
A PDB limits how many pods can be voluntarily evicted at once during maintenance. Anti-affinity controls where pods are scheduled in the first place. They solve different problems, and mature clusters use both together.
Does Kubernetes version skew really matter for small teams?
Yes, because the failure mode is not gradual. An unsupported skew between kubelet and API server can produce node registration failures or rejected manifests with no obvious cause. Upgrading one minor version at a time is slower but far cheaper than debugging a broken control plane.
Can network policies replace a service mesh?
Not entirely. NetworkPolicy handles L3/L4 allow-and-deny rules between pods. A service mesh adds identity-based authorization, mTLS, and traffic shaping at L7. Many teams start with policies and add a mesh only when they need those higher-level controls.
