GitOps promises a clean, auditable path from a Git commit to a running cluster. Yet nearly every team that adopts it eventually hits the same wall: the pipeline looks green, the cluster looks wrong, and nobody knows why. Industry surveys suggest that while most organizations now run GitOps in production, a large share report that drift, secrets, and multi-environment sprawl are still unsolved. In other words, the tooling matured faster than the operating model around it. (GitOps)
This article walks through the problems that show up most often, why they happen, and what actually works in practice. It is written for platform engineers, SREs, and engineering leads who have moved past “hello world” and are now fighting the real thing. (Openai 100 math problems solved 24 days 2026)
Drift and Reconciliation Loops That Never Converge
The single most common complaint is configuration drift — the gap between what Git says and what the cluster actually runs. Someone runs a quick `kubectl edit` during an incident, forgets to commit it, and three weeks later a deploy wipes the fix. Or worse, it does not wipe it, and two sources of truth quietly disagree.
The subtler version is the reconciliation loop that fights itself. Two controllers both want to own the same field, so each reconcile undoes the other. The sync status flips between healthy and out-of-sync every few minutes, alert fatigue sets in, and people start ignoring the dashboard. That is the real damage: not the drift itself, but the loss of trust in the signal.
What works in practice:
- Make the cluster read-only for humans. Remove broad `edit` rights and route emergency changes through a documented break-glass process that forces a commit afterward.
- Turn on drift detection and treat it as a first-class alert, not noise.
- Audit which controllers own which fields before adding a new operator. Field ownership conflicts are almost always a design problem, not a tool bug.
A useful sanity check is whether your team could rebuild the cluster from Git alone. If the answer is “mostly,” you have drift you have not found yet. The same discipline of verifying outcomes rather than trusting a green checkmark shows up in other engineering domains too — a claimed result means little until someone independently checks the work.
Secrets in Git: The Problem Nobody Fully Solves
Storing secrets in Git is the classic GitOps contradiction. Git is designed to be permanent and distributed; secrets are supposed to be ephemeral and tightly scoped. Teams usually land on one of four patterns:
| Approach | How it works | Main drawback |
|---|---|---|
| Sealed Secrets | Encrypts secrets so only the cluster can decrypt | Key rotation is painful; re-encrypting on cluster rebuild |
| SOPS | Encrypts values inside the file, keys managed externally | Key management becomes its own project |
| External Secrets Operator | Pulls from Vault or cloud secret managers at runtime | Adds a dependency; sync failures block deploys |
| Plain env vars in CI | Simple, fast to set up | Secrets leak into logs and history |
The mistake teams make most often is picking a tool before deciding who owns rotation. Encryption at rest in Git does not help if the decryption key sits in the same repository’s CI variables. Decide the rotation owner first, then pick the tool that fits that owner’s workflow.
Multi-Environment Sprawl and Promotion Pain
Dev, staging, and production are supposed to differ only in values. In reality, the repositories diverge. Someone copies a folder, tweaks it, and now the environments are maintained by hand. Kustomize overlays and Helm value files help, but they do not stop duplication — they just make it tidier.
The symptom to watch for is environment drift inside the repo itself. When the staging overlay has a resource that production does not, you no longer have a promotion pipeline; you have three separate projects that happen to share a name.
Practical fixes that hold up:
- Keep one base and make every environment difference explicit in a single overlay file.
- Add a CI check that diffs rendered manifests between environments and fails on unexpected divergence.
- Promote by changing an image tag or a reference, never by copying files.
Teams that skip this step usually discover the problem during an incident, when they need production to match staging and it does not.
Flux vs. Argo CD: The Trade-Offs People Underestimate
Both are mature, both are CNCF projects, and both will run your cluster. The differences that bite are operational, not feature-level.
Argo CD offers a strong UI, which helps during onboarding and debugging, and a clear application-centric model. The cost is that the UI becomes a crutch, and multi-tenancy requires careful project scoping.
Flux is more composable and CLI-first, which suits platform teams that want everything in Git. The cost is a steeper learning curve and less immediate visibility for developers who are not comfortable reading controller logs.
The mistake is choosing based on a feature checklist. Choose based on who will debug it at 2 a.m. If developers self-serve deployments, a UI helps. If a small platform team owns the pipeline, composability wins. Either way, budget real time for the learning curve — most failed adoptions are staffing problems wearing a tooling costume.
The Gaps GitOps Does Not Cover
GitOps is excellent at declarative, steady-state workloads. It is awkward for everything else. Common gaps include:
- One-off migrations and data backfills that must run once, in order, with a human watching.
- Imperative bootstrap steps — the chicken-and-egg problem of installing the GitOps controller itself.
- Stateful workloads where a database failover needs a runbook, not a reconcile loop.
- Cross-cluster ordering, where resource A in cluster one must exist before resource B in cluster two.
The honest answer is that GitOps covers maybe 80 percent of the surface. The remaining 20 percent needs explicit runbooks, and pretending otherwise creates a false sense of safety. Document the imperative steps. Put them in version control as scripts even if they are not reconciled.
Repository Sprawl and Unclear Ownership
A typical setup ends up with an application repo, a config repo, an environment repo, and a platform repo. Each has its own reviewers, its own CI, and its own idea of what “done” means. Pull requests stall because nobody is sure who approves a change to the shared base.
What actually helps is boring: publish an ownership map (a `CODEOWNERS` file is a good start), cap the number of repos, and give each one a single accountable team. Fewer repos with clear owners beat a perfect structure that nobody owns.
Silent Sync Failures and Observability Gaps
The most dangerous state in GitOps is not “failed.” It is “unknown.” A controller that stops reconciling without alerting is worse than one that errors loudly, because everyone assumes the pipeline is working.
Three alerts are worth having from day one:
- Last successful sync age — alert if any app has not reconciled in longer than its expected interval.
- Drift count — alert when the number of out-of-sync resources crosses a threshold.
- Controller health — the GitOps controller itself needs liveness and error-rate monitoring.
Beyond alerting, track time to detect and time to recover for sync failures. If nobody can answer “how long was production out of sync last month,” you have an observability gap, not a tooling gap.
Security of the Pipeline Itself
GitOps moves trust into the pipeline, which means the pipeline becomes the target. Over-privileged CI credentials, a controller with cluster-admin rights, and unsigned commits are all common. A compromised CI token in a GitOps setup can do far more damage than in a traditional deploy, because the controller is designed to apply whatever Git says. (What is GitOps?)
Practical hardening:
- Scope the controller’s permissions per namespace or per project, not cluster-wide.
- Require signed commits and protected branches on the config repo.
- Rotate CI credentials on a schedule and treat them as production secrets.
- Review the full set of resources the controller can reach, beyond what it currently does.
Supply-chain security frameworks exist precisely because pipeline trust is hard to get right; treat the GitOps pipeline as part of your supply chain, not as infrastructure that sits outside it.
Conclusion
GitOps is not a product you install; it is an operating discipline you maintain. The problems that matter — drift, secrets, environment sprawl, unclear ownership, silent failures, and pipeline security — are all organizational as much as technical. The teams that succeed are the ones that treat the Git repository as the single source of truth and invest in the human processes around it: clear ownership, honest documentation of the gaps, and alerting that people actually trust. Start with drift detection and ownership maps. They are unglamorous, cheap, and they prevent most of the fires that come later.
Frequently Asked Questions
What is the most common GitOps problem?
Configuration drift is the most frequently reported issue. A manual change to a live cluster creates a second source of truth, and the Git-declared state slowly stops matching reality. Drift detection and read-only cluster access for humans solve most of it.
How do teams handle secrets in GitOps?
Most teams use Sealed Secrets, SOPS, or an external secret manager such as Vault or a cloud provider’s service. The tool matters less than deciding who owns key rotation before you start. Secrets committed as plain environment variables are the pattern to avoid.
Should we choose Flux or Argo CD?
It depends on who will operate it. Argo CD’s UI suits teams where developers self-serve deployments. Flux’s composable, CLI-first model suits platform teams that want everything declared in Git. Both are mature CNCF projects; the learning curve, not the feature list, is usually the deciding factor.
Why does my GitOps sync keep flipping between healthy and out-of-sync?
Two controllers are likely fighting over the same field. Check which operators manage the affected resource and audit field ownership. Flipping status is a design conflict, not a bug in the GitOps tool, and it erodes trust in your dashboards faster than almost anything else.
Does GitOps work for databases and stateful workloads?
Partially. GitOps handles the declarative parts well — the StatefulSet, the service, the config. It handles failover, data migrations, and one-off backfills poorly. Keep those in documented runbooks and version-controlled scripts rather than forcing them into a reconcile loop.
How do we know if our GitOps setup is actually working?
Ask three questions: can you rebuild the cluster from Git alone, how long was production out of sync last month, and who approves a change to the shared base? If any answer is unclear, you have a gap worth fixing before adding more tooling.
Final Takeaway
The pattern across every problem above is the same: GitOps automates applying intent, not defining it. A repository full of drift, ambiguous ownership, and unrotated keys will simply be applied more consistently. Treat the discipline as an ongoing maintenance commitment rather than a one-time migration, and revisit these failure modes on a regular cadence — quarterly reviews of drift alerts, ownership maps, and sync latency catch most regressions before an incident does. The tooling will keep evolving, but the operating habits that make GitOps reliable are stable, and they are worth building early.
