In 2023, the Cloud Native Computing Foundation reported that a large share of organizations running Kubernetes had adopted GitOps or planned to within a year. That number keeps climbing, and it is easy to see why. When Git becomes the single source of truth for your infrastructure, every change is reviewable, reversible, and documented by default. But here is the catch that trips up a lot of teams: adopting a GitOps tool is not the same as following GitOps best practices. Plenty of organizations install Argo CD or Flux, point it at a repo, and then quietly reintroduce the same chaos they were trying to escape. This article walks through what actually works in practice, where teams stumble, and how to build a workflow that stays healthy as your cluster count grows.
Start With the Core Principle and Don’t Drift From It
The whole idea rests on one rule: the desired state of your system lives in Git, and an automated agent continuously makes reality match it. That is the definition the OpenGitOps working group settled on, and it is worth internalizing before you touch any tooling.
In practice, teams violate this principle in small, innocent ways. Someone runs a quick `kubectl edit` to fix a broken pod at 2 a.m. A contractor applies a manifest by hand. A CI pipeline pushes credentials straight into the cluster. Each of these seems harmless in the moment, but they break the guarantee that Git tells the truth about production.
The fix is cultural as much as technical. Treat manual changes to production as incidents, not conveniences. If a hotfix is genuinely urgent, make the change in Git first, even if that means a fast-tracked pull request. The five minutes you save by editing live are the five minutes that make your next debugging session a nightmare.
A practical tip: enable drift detection and alert on it. Both major GitOps agents can report when live state diverges from Git. Wire those alerts into the same channel your team already watches, and treat each one as a small teaching moment rather than noise.
Structure Your Repositories Before You Scale
Repository layout is one of the genuinely contested topics in the GitOps world, and there is no universal answer. What matters is that you choose deliberately and document the reasoning.
| Layout | Best for | Watch out for |
|---|---|---|
| Single monorepo | Small teams, tight coupling | Merge conflicts, slow CI, blast radius |
| App and config repos split | Most growing teams | Coordination overhead between repos |
| Repo per environment | Strict isolation, regulated industries | Duplication, drift between environments |
A common and durable pattern is to keep application source code separate from deployment manifests. The app repo holds your code and builds images. The config repo holds the Kubernetes manifests, Helm values, or Kustomize overlays that reference those images. When a new image is built, a bot or pipeline updates the config repo with a pull request. The GitOps agent then reconciles the cluster.
This separation gives you a clean audit trail. You can see exactly which commit changed which environment, and rolling back is a single revert. It also means a developer without cluster credentials can still ship code, because merging to the config repo is the deployment mechanism.
Keep Environment Config Explicit
Avoid the temptation to make one giant set of manifests that “works everywhere” through a tangle of conditionals. Overlays and values files that clearly state what differs between staging and production are far easier to reason about. When something breaks in production, you want to read the config and understand it in seconds, not trace inheritance chains.
Make Every Change Flow Through Pull Requests
If merging to a branch equals deploying, then your pull request process is your deployment process. That reframing changes how you should treat it.
Protect your main branches. Require reviews. Require passing checks. The people who can merge to the deployment branch effectively hold deploy rights, so scope that permission as carefully as you would scope cluster admin access. This is one of the most overlooked security controls in GitOps, and it connects directly to broader AI governance principles, risks, and best practices thinking about least privilege and auditable change control, even if your workloads are not AI-related. (DevOps)
A few habits that pay off:
- Keep pull requests small and single-purpose. A PR that bumps one image is trivial to review and revert.
- Automate the boring parts. Let a bot open the image-update PR so humans review, not author.
- Require a diff preview. Tools that show the rendered manifest diff before merge catch a surprising number of mistakes.
- Never commit plaintext secrets. Use a sealed-secrets tool, an external secrets operator, or a vault integration.
That last point deserves emphasis. Committing a Kubernetes Secret in plain YAML is one of the most common GitOps mistakes, and it is a serious one. Once a secret lands in Git history, it is effectively public to anyone with repo access, and scrubbing history is painful. Choose a secrets strategy early and enforce it with automated scanning.
Prefer Pull-Based Reconciliation Over Push
Older CI-driven deployment models push changes into the cluster, which means your CI system needs powerful credentials to your production environment. That is a large attack surface sitting outside your cluster.
Pull-based GitOps flips this. An agent runs inside the cluster, watches Git, and pulls changes when they appear. The cluster reaches out; nothing reaches in. Your CI system never needs production credentials, and your firewall can stay closed to inbound deployment traffic.
This is the model both Argo CD and Flux use, and it is the reason GitOps pairs so naturally with hardened cluster security. If you are still pushing manifests from CI, migrating to a pull-based agent is one of the highest-value changes you can make.
Common mistake: running the GitOps agent with cluster-admin rights “just to get it working.” Scope its permissions to what it actually manages. A compromised agent with god-mode access is a very bad day.
Pair GitOps With Progressive Delivery
GitOps handles what gets deployed. It does not, by itself, control how a rollout proceeds. That is where progressive delivery comes in.
Instead of flipping 100% of traffic to a new version at once, canary and blue-green strategies shift traffic gradually and watch metrics along the way. If error rates spike, the rollout halts or rolls back automatically. The combination is powerful: Git declares the desired end state, and a rollout controller manages the safe path to reach it.
In practice, teams often start with simple rolling updates, then add canary analysis once they have solid metrics and alerting. Do not skip straight to sophisticated rollout automation before you can reliably measure whether a deployment is healthy. Progressive delivery without good observability is just a slower way to fail.
Watch for Drift and Reconcile Continuously
Drift is what happens when the live cluster and Git disagree. It creeps in through manual changes, failed partial applies, or external controllers that mutate resources. Left alone, drift erodes trust in your source of truth.
Set your agent to reconcile on a schedule, not just when commits land. A periodic sync catches changes that slipped in between deployments. Then alert on any resource that stays out of sync, because a persistent drift usually signals a real problem: a misconfigured controller, a permissions gap, or a human bypassing the process.
It also helps to distinguish between expected and unexpected drift. Some controllers legitimately modify resources after creation, such as autoscalers adjusting replica counts. Configure your agent to ignore those specific fields so you do not drown in false alarms.
Build the Right Team Habits
Tooling gets the headlines, but habits decide whether GitOps holds up over time. A few that separate mature teams from struggling ones:
- Review config changes like code. They are code. Give them the same scrutiny.
- Keep the feedback loop short. Slow reconciliation or unclear sync status kills confidence. Invest in dashboards that show what is deployed where.
- Document your layout. New hires should understand the repo structure in an afternoon, not a month.
- Run game days. Practice rolling back a bad deploy through Git. You want that muscle memory before you need it.
- Audit access regularly. Revisit who can merge to protected branches and who can administer the agent.
There is a governance angle here too. As systems grow, the discipline of traceable, reviewable change becomes a compliance asset, not just an engineering nicety. Teams thinking through AI governance principles, risks, and best practices for automated decision-making will recognize the same pattern: clear ownership, auditable history, and least-privilege access. GitOps gives you all three almost for free, if you resist the shortcuts.
Conclusion
GitOps works best when you treat it as a discipline rather than a plugin. The tools are mature and the patterns are well understood, but the wins come from the boring stuff: protecting your branches, keeping secrets out of Git, preferring pull over push, and treating manual production edits as incidents. Get those fundamentals right and the rest compounds. Your deployments become reversible, your audit trail writes itself, and your team spends less time firefighting and more time shipping. Start with one repository and one cluster, prove the workflow, then expand. The organizations that succeed with GitOps are rarely the ones that moved fastest. They are the ones that stayed consistent.
Frequently Asked Questions
What is the single most important GitOps best practice?
Keep Git as the only source of truth and never let manual changes to production stand. Every other practice, from pull requests to drift detection, depends on this one holding true. If live state and Git can quietly diverge, the whole model loses its value.
Should application code and deployment config live in the same repository?
It depends on team size and structure, but most growing teams benefit from separating them. Keeping app source and deployment manifests apart gives you a cleaner audit trail, simpler rollbacks, and lets developers ship without direct cluster access. Smaller teams can start with one repo and split later.
How do I handle secrets in a GitOps workflow?
Never commit plaintext secrets to Git. Use a dedicated approach such as sealed secrets, an external secrets operator that pulls from a vault, or encrypted files managed with a key. Pick one strategy early, enforce it with automated scanning, and document it so nobody reintroduces plaintext secrets later.
Is pull-based deployment really better than push?
For most production setups, yes. Pull-based agents run inside the cluster and reach out to Git, so your CI system never needs production credentials and you can keep inbound deployment traffic closed. Push models still work, but they widen your attack surface for little benefit.
How often should the GitOps agent reconcile?
Set a periodic reconciliation schedule in addition to syncing on every commit. A regular sync catches drift that appears between deployments, and alerting on persistent out-of-sync resources surfaces real problems early. Tune the interval to your change frequency, but avoid disabling periodic sync entirely.
Can GitOps work for teams that are not on Kubernetes?
Yes. The core principle, Git as the source of truth with automated reconciliation, applies to virtual machines, serverless platforms, and even network configuration. Kubernetes is simply where the tooling is most mature. The habits around review, auditability, and drift detection transfer directly.
Common mistake: enabling automated sync without automated pruning. Resources removed from Git linger in the cluster indefinitely, quietly consuming capacity and obscuring what is actually running. Turn on pruning deliberately, and pair it with sync windows so a bad commit cannot wipe production during peak hours.
Finally, measure your GitOps practice the way you measure any other system. Track sync duration, drift frequency, rollback time, and the share of changes that arrive through pull requests rather than manual edits. These numbers tell you whether the discipline is holding or quietly eroding. When manual edits creep back up, treat it as an early warning and investigate the friction that pushed people around the process in the first place.
