Roughly half of all production Kubernetes incidents trace back to a handful of repeatable misconfigurations, not exotic bugs. That single observation explains why experienced platform teams spend more time on a short checklist than on deep debugging. The same failures show up again and again: pods that won’t start, services that won’t connect, and clusters that quietly degrade under pressure. This guide walks through the most common Kubernetes problems, what actually causes them, and the practical steps that resolve them without guesswork. (Openai 100 math problems solved 24 days 2026) (Kubernetes)
Why the Same Kubernetes Problems Keep Appearing
Kubernetes is a declarative system. You describe a desired state, and controllers work to make reality match it. That design is powerful, but it also means small mistakes in your manifests propagate silently until something breaks loudly. A missing resource request, a typo in a label, or an outdated API version can sit dormant for weeks before surfacing as an outage.
The pattern is familiar to anyone who has managed a cluster upgrade. Teams that plan the upgrade carefully, check deprecated APIs, and test in staging hit far fewer surprises than teams that click “upgrade” and hope. If you want a structured approach, this walkthrough of the EKS cluster upgrade to Kubernetes 1.37 in 13 steps covers the sequencing that prevents most upgrade-related breakage. (Kubernetes Documentation: Concepts)
The cluster doesn’t fail because it’s complicated. It fails because one line was wrong and nobody noticed until traffic arrived.
Pods Stuck in CrashLoopBackOff
CrashLoopBackOff is the most visible Kubernetes problem. A container starts, exits, restarts, and repeats. The kubelet applies an exponential backoff delay between restarts, so the gap between attempts grows over time. The pod status tells you that it’s crashing; it never tells you why.
The fix starts with logs. Running `kubectl logs –previous` shows output from the last crashed instance, which is usually where the real error lives. Common causes include:
- A missing environment variable or config file the application expects at startup.
- A bad command or entrypoint in the container spec that exits immediately.
- A failed dependency check where the app refuses to boot because a database isn’t reachable yet.
- A liveness probe that is too aggressive, killing a container that was still starting up.
That last one catches people constantly. A liveness probe with a two-second initial delay on an app that needs thirty seconds to warm up will kill the container before it ever becomes healthy. Practical tip: set `initialDelaySeconds` generously and use a startup probe for slow-booting applications. In practice, most CrashLoopBackOff incidents resolve once you read the previous container’s logs rather than the current one’s.
ImagePullBackOff and ErrImagePull
These two statuses mean the node cannot fetch the container image. The distinction matters slightly: `ErrImagePull` is the immediate failure, and `ImagePullBackOff` is the retry state with increasing delay.
The causes are narrow and predictable:
- Wrong image name or tag. A typo in the registry path or a tag that doesn’t exist.
- Private registry without credentials. The pod needs an `imagePullSecrets` reference pointing to a valid pull secret.
- Registry rate limits. Public registries throttle anonymous pulls, which bites hard during a large rollout.
- Network or firewall restrictions blocking the node from reaching the registry.
A quick check is `kubectl describe pod ` and reading the Events section. The error message is usually explicit, something like “manifest unknown” or “unauthorized.” A common mistake is creating the pull secret in the wrong namespace; secrets don’t cross namespace boundaries, so a secret in `default` won’t help a pod in `production`.
OOMKilled: When Memory Limits Bite
When a container exceeds its memory limit, the kernel terminates it and Kubernetes reports the reason as OOMKilled. This is one of the most misunderstood problems because the fix isn’t always “raise the limit.”
Two scenarios dominate. First, the limit is genuinely too low for normal operation, and the app gets killed during a traffic spike. Second, the application has a memory leak, and raising the limit only delays the inevitable crash.
Practical approach: check `kubectl describe pod` for the exit code (137 indicates a SIGKILL, consistent with OOM). Then compare actual usage against the limit using metrics. If usage climbs steadily over hours, it’s a leak. If it spikes briefly and recovers, the limit is simply too tight. Raising a limit to hide a leak is the most expensive shortcut in Kubernetes.
Pending Pods That Never Schedule
A pod stuck in Pending hasn’t been assigned to a node. The scheduler couldn’t find a fit. The reasons are almost always one of these:
| Cause | How to Confirm | Typical Fix |
|---|---|---|
| Insufficient CPU or memory | `kubectl describe pod` shows “Insufficient cpu” | Reduce requests or add nodes |
| Taints without tolerations | Events mention taint mismatch | Add toleration or remove taint |
| Node affinity too strict | Events mention node affinity | Relax the selector |
| Unbound PersistentVolumeClaim | PVC status is Pending | Fix storage class or provisioner |
| ResourceQuota exceeded | Events mention quota | Raise quota or lower requests |
The single most common contributor is missing resource requests. When a pod declares no requests, the scheduler can’t reason about capacity properly, and clusters end up overcommitted in ways nobody planned. Always set requests. Limits are a separate decision, but requests are non-negotiable for predictable scheduling.
Evicted Pods and Node Pressure
Kubernetes evicts pods when a node runs short on a critical resource: memory, disk, or process IDs. Unlike a crash, an eviction is the system protecting itself. Running `kubectl describe node` reveals conditions like `MemoryPressure` or `DiskPressure`.
Disk pressure is the sneaky one. Container logs, unused images, and emptyDir volumes accumulate until the kubelet starts reclaiming. Teams often discover this only after a batch of pods disappears overnight. Practical tip: set log rotation at the container runtime level and monitor node disk usage as a first-class metric, not an afterthought.
Services That Won’t Connect
A service exists, pods are running, and yet requests fail. This is one of the most frustrating Kubernetes problems because everything looks healthy. The debugging path is short if you follow it in order:
- Check endpoints. `kubectl get endpoints ` should list pod IPs. If it’s empty, the service selector doesn’t match any pod labels. This is the number one cause.
- Check targetPort. The service’s `targetPort` must match the port the container actually listens on, not the port you think it listens on.
- Check NetworkPolicy. A default-deny policy with no allow rule will silently drop traffic.
- Check readiness. Pods that fail readiness probes are removed from endpoints, so the service has nothing to route to.
Label mismatches deserve emphasis. A service selecting `app: web` won’t find pods labeled `app: web-app`. That single character difference has caused more outages than any exotic networking bug.
DNS Resolution Failures Inside the Cluster
When pods can’t resolve service names, the problem usually sits with CoreDNS, the cluster’s DNS provider since Kubernetes 1.13. Symptoms include timeouts on internal hostnames and applications that work fine externally but fail on internal calls.
Common causes include CoreDNS pods being unhealthy, `ndots` misconfiguration causing excessive search-domain lookups, and NetworkPolicy rules that block DNS traffic on port 53. A quick test is launching a temporary debug pod and running a DNS lookup against a known service name. If that fails, inspect CoreDNS pod logs and resource limits; CoreDNS starving for memory is a surprisingly frequent culprit.
RBAC Forbidden Errors
RBAC errors appear as `Forbidden` responses from the API server. They’re common after a deployment introduces a new service account, or after someone tightens permissions.
The error message is actually helpful here. It names the user, the verb, and the resource. Reading it carefully usually reveals the gap: a Role exists but the RoleBinding points to the wrong subject, or the permission is namespaced when the request is cluster-scoped. A frequent oversight is granting access to a service account in one namespace while the workload runs in another. Always verify the namespace on both the binding and the workload.
Deprecated and Removed APIs After Upgrades
Upgrades break workloads when manifests reference API versions that no longer exist. For example, `extensions/v1beta1` Ingress was removed in Kubernetes 1.22, released in August 2021. Clusters that skipped the deprecation warnings went straight to failure.
The practical defense is tooling that scans manifests for removed APIs before the upgrade, plus a staging cluster that mirrors production versions. This is exactly why structured upgrade planning matters, and the same discipline applies to AI-driven engineering workflows where automation multiplies both speed and mistakes, as seen in coverage of AI creating more work for engineering teams. Automation helps, but only when the underlying manifests are clean.
Resource Requests, Limits, and Noisy Neighbors
Missing limits let one workload consume a node’s resources and starve its neighbors. Missing requests let the scheduler overcommit. Both are configuration problems, and both are preventable.
A reasonable starting policy: set requests based on observed steady-state usage, set memory limits close to requests, and be more generous with CPU limits (or omit them entirely, since CPU is compressible and throttling is gentler than termination). Review these numbers quarterly. Workloads change, and yesterday’s sizing becomes today’s incident.
Conclusion
The common Kubernetes problems are not mysterious. CrashLoopBackOff, ImagePullBackOff, OOMKilled, Pending pods, evictions, service failures, DNS issues, RBAC errors, and deprecated APIs account for the overwhelming majority of production pain. Each has a documented cause and a short diagnostic path. The teams that suffer least are not the ones with the most sophisticated tooling. They’re the ones who set resource requests properly, read logs before guessing, check endpoints before blaming the network, and plan upgrades instead of improvising them. Kubernetes rewards discipline far more than it rewards cleverness.
Frequently Asked Questions
What is the most common Kubernetes problem in production?
CrashLoopBackOff and service connectivity failures top the list. Both stem from configuration mistakes rather than platform bugs, which is why they’re so preventable. Reading previous container logs and checking service endpoints resolves most cases within minutes.
How do I debug a pod that won’t start?
Start with `kubectl describe pod` to read the Events section, then `kubectl logs –previous` to see the last crashed container’s output. Those two commands answer the majority of “why won’t this start” questions before you need anything more advanced.
Why does my Kubernetes service have no endpoints?
Almost always a label selector mismatch. The service selector must exactly match pod labels, including case and spelling. Run `kubectl get endpoints ` to confirm, then compare the selector against `kubectl get pods –show-labels`.
What causes OOMKilled and how do I fix it?
The container exceeded its memory limit and the kernel terminated it. Check whether usage climbs steadily (a leak) or spikes briefly (a tight limit). Fix leaks in code; raise limits only when the application’s normal usage genuinely requires more memory.
How do I prevent problems during a Kubernetes upgrade?
Scan manifests for removed API versions before upgrading, test in a staging cluster that mirrors production, and upgrade one minor version at a time. Most upgrade failures come from deprecated APIs that were flagged in release notes months earlier.
Is it better to set CPU limits or leave them off?
Many teams leave CPU limits off and rely on requests for scheduling fairness, since CPU throttling degrades latency more gradually than memory termination. Memory limits, by contrast, should almost always be set to prevent a single workload from destabilizing a node.
