Kubernetes turned ten years old as a public project in June 2024, and in that decade it went from an internal Google tool to the default operating system of the cloud. Yet the same survey data that shows near-universal adoption also shows a stubborn pattern: a large share of production incidents trace back to a handful of avoidable configuration mistakes. Misconfigured resource limits, missing probes, and cluster-admin credentials handed out like candy are still among the most common causes of outages. The good news is that the fixes are well documented and rarely exotic. This guide walks through the practices that separate clusters that quietly hum along from clusters that page someone at 3 a.m. (Configuration Best Practices)
Start With Resource Requests and Limits
If there is one habit that pays for itself fastest, it is setting resource requests and limits on every container. Requests tell the scheduler how much CPU and memory a pod needs to be placed on a node. Limits cap what it can consume. Get these wrong and you get two classic failure modes: pods stuck in Pending because no node has room, or a single runaway container starving its neighbors.
A practical approach that works in real teams:
- Deploy without limits for a short observation window in staging and watch actual usage.
- Set requests close to the observed 90th percentile, not the peak.
- Set memory limits equal to or slightly above requests so the pod lands in the Guaranteed or Burstable QoS class rather than BestEffort.
- Revisit quarterly, because workloads drift.
The most common mistake is copying limits from a tutorial. A Java service with a 512Mi limit will spend its life in OOMKilled purgatory because the JVM heap alone often needs more than that. Another frequent error is setting CPU limits very low “to save money” and then wondering why latency spikes under load, since CPU throttling is silent and brutal.
Use Probes Deliberately, Not Reflexively
Liveness, readiness, and startup probes are among the most misunderstood features in Kubernetes. The Kubernetes documentation on the pod lifecycle explains the distinction clearly, but in practice teams often bolt a liveness probe onto everything and move on.
Here is what actually works:
- Readiness probes should check whether the app can serve traffic, including downstream dependencies it truly needs. This is the probe that removes a pod from a Service endpoint.
- Liveness probes should only answer one question: is this process so broken that restarting it is the only fix? Pointing a liveness probe at a database check is a classic self-inflicted outage, because a brief database blip restarts every pod at once.
- Startup probes protect slow-booting applications from being killed before they finish initializing.
A rule of thumb many SRE teams use: if you cannot explain what a failed liveness probe would tell you, delete it. Readiness alone plus good alerting covers most cases.
Keep Manifests in Git and Deploy Through GitOps
Manual `kubectl apply` from a laptop is fine for learning and terrible for production. GitOps treats a Git repository as the single source of truth for cluster state, with a controller continuously reconciling the live cluster against what is committed. Argo CD and Flux are the two dominant tools, and both are CNCF projects with mature communities.
The benefits are concrete rather than philosophical. Every change has an author, a timestamp, and a diff. Rollbacks become a `git revert`. Drift gets detected automatically instead of discovered during an incident. Teams that adopt GitOps consistently report fewer “who changed this?” conversations, which is worth more than it sounds.
A practical tip from teams running this in production: keep environment-specific configuration in separate overlays or directories rather than long chains of branches. Branch-per-environment sounds tidy but turns merges into a nightmare within a month.
Lock Down Access With Namespaces and RBAC
Role-Based Access Control is only as good as the roles you write. The single most common anti-pattern is binding users or service accounts to `cluster-admin` because “it’s just for now.” That binding outlives the person who created it roughly 100 percent of the time.
A workable baseline:
| Practice | Why it matters |
|---|---|
| One namespace per team or application | Limits blast radius and simplifies quotas |
| Namespace-scoped Roles instead of ClusterRoles | Prevents accidental cluster-wide changes |
| Separate service accounts per workload | Stops one compromised pod from impersonating another |
| No long-lived tokens in CI | Use short-lived credentials via workload identity |
Pair this with NetworkPolicy so that pods can only talk to what they need to talk to. Default-deny ingress in each namespace, then open specific paths. It takes an afternoon to set up and prevents an entire category of lateral-movement attacks.
Apply Pod Security Standards
The old PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25, replaced by the built-in Pod Security Standards: Privileged, Baseline, and Restricted. The Pod Security Standards documentation lays out exactly what each level enforces, and the Restricted profile is a sensible target for most application workloads.
In practice, enabling Restricted cluster-wide on day one breaks things. A staged rollout works better: run the policy in `warn` mode first, collect the warnings from your workloads for a couple of weeks, fix the manifests, then switch to `enforce`. Teams that skip the warning phase usually end up disabling the policy entirely after the first broken deployment, which is worse than never having tried.
Pin Image Versions and Scan Them
Using the `latest` tag in production is one of those choices that feels convenient until a deployment silently pulls a different image than the one that was tested. The Kubernetes configuration best practices page recommends avoiding `latest` for exactly this reason.
Better habits:
- Pin to a specific version tag, and ideally to an image digest for critical services.
- Scan images in CI as well as in the registry, so vulnerabilities block the build rather than the deploy.
- Rebuild base images on a schedule even when your code has not changed, because CVEs do not care about your release cadence.
A subtle mistake here is pinning the tag but not the digest in environments where the registry allows tag overwrites. If your registry permits mutable tags, an attacker with push access can swap the image under you.
Design for Failure With Multiple Replicas and Disruption Budgets
A single-replica Deployment is a single point of failure wearing a Kubernetes costume. Run at least two replicas of anything user-facing, spread across nodes with pod anti-affinity or topology spread constraints so a single node failure does not take out every copy.
Then add a PodDisruptionBudget so voluntary disruptions like node drains during upgrades do not take the whole service down. This matters more than people expect: a cluster upgrade that drains nodes one at a time is safe only if your PDBs and replica counts cooperate. If you are planning one, the walkthrough on the EKS cluster upgrade to Kubernetes 1.37 in 13 steps covers the sequencing and the drain-safety checks in detail. (Kubernetes)
Automate Policy and Governance Early
As clusters multiply, manual review stops scaling. Policy engines such as Open Policy Agent’s Gatekeeper or Kyverno let you encode rules as code: no `latest` tags, mandatory resource limits, required labels, no privileged containers. This is a form of AI governance thinking applied to infrastructure, and the same principles of documented risk, clear ownership, and auditable decisions apply. A useful primer on that broader discipline is this overview of AI governance principles, risks, and best practices, which maps surprisingly well onto cluster policy design.
The practical tip: start with three policies, not thirty. Enforce the ones that prevent outages first, then expand.
Watch the Cost and Complexity Creep
Kubernetes makes it easy to request resources you never use. Idle namespaces, over-provisioned requests, and forgotten load balancers quietly accumulate. Tools like the Kubernetes resource metrics pipeline, combined with cost allocation by namespace or label, turn this from guesswork into a report.
There is also a human cost. A 2026 Anthropic study on AI-assisted work found that automation often shifts effort rather than removing it, and platform tooling follows the same pattern: a self-service cluster platform reduces tickets but creates a new job called “platform engineering.” That trade is usually worth it, but it should be a deliberate choice. The findings are summarized in this piece on AI creating more work: the Uber, ChatGPT, and Anthropic study, and the lesson transfers directly to infrastructure automation.
Document Ownership and On-Call Reality
Every namespace should have an owner listed somewhere a stranger can find it. Every alert should route to a team that knows what to do with it. This sounds like bureaucracy until the first incident where nobody knows who owns the failing service, and then it sounds like the most important document in the company.
A lightweight pattern that works: a `OWNERS` file or annotation in each application repository, plus a runbook link in the alert payload. No wiki archaeology required at 3 a.m.
Conclusion
Kubernetes rewards discipline more than cleverness. The practices that matter most are unglamorous: honest resource requests, probes that mean something, Git as the source of truth, least-privilege access, pinned images, and policies that catch mistakes before they reach production. None of them require a platform team of twenty. What they require is consistency, and a willingness to fix the boring things before they become the urgent things. Start with resource limits and RBAC this week, add GitOps next month, and the cluster you run a year from now will be dramatically calmer than the one you inherited.
Frequently Asked Questions
What are the most important Kubernetes best practices for beginners?
Start with four things: set resource requests and limits on every container, add readiness probes, pin image versions instead of using `latest`, and avoid granting `cluster-admin` to anyone who does not absolutely need it. These four habits prevent the majority of early production incidents. Everything else, including GitOps and policy engines, can follow once the basics are consistent.
How many replicas should a production Deployment have?
At least two for anything user-facing, and three if you can afford it. Two replicas survive a single node failure; three survive a node failure during a rolling update without dropping below your disruption budget. Pair the replica count with pod anti-affinity or topology spread constraints, otherwise the scheduler may place all replicas on the same node and defeat the purpose.
Should liveness probes check database connectivity?
No. A liveness probe that fails when the database is briefly unavailable will restart every pod of your service at once, turning a small blip into a full outage. Liveness probes should only detect a process that cannot recover on its own. Dependency health belongs in readiness probes, where a failure removes the pod from the Service endpoint without killing it.
What replaced PodSecurityPolicy in Kubernetes?
The Pod Security Standards replaced PodSecurityPolicy, which was deprecated in 1.21 and removed in 1.25. The standards define three levels: Privileged, Baseline, and Restricted. Most application workloads should target Restricted, but roll it out in warn mode first so you can see which manifests need fixing before enforcement starts blocking deployments.
Is GitOps required for production Kubernetes?
It is not strictly required, but it is strongly recommended once more than one person deploys to a cluster. GitOps gives you an auditable history of every change, makes rollbacks a single revert, and detects configuration drift automatically. Argo CD and Flux are both mature CNCF projects, and either will do the job. The main requirement is discipline: nobody applies manifests by hand, ever.
How do I stop Kubernetes costs from creeping up?
Set requests based on observed usage rather than guesses, delete idle namespaces and unused load balancers on a schedule, and label everything so cost can be attributed to a team. Review resource requests quarterly, because workloads drift and a request that was accurate six months ago is often double what is needed today. Cost visibility by namespace is usually the single highest-leverage change.
