Quick Answer: What are the most common mistakes when using terraform in production?
The most common mistakes when using Terraform in production cluster around state management, change control, and access. Teams lose or corrupt state by running with local backends or without locking, apply changes without a saved and reviewed plan, commit secrets into version control, and let manual console edits create drift that nobody detects.
The scope is broader than a single misconfiguration. Some problems are infrastructure choices, like splitting one giant root module into smaller states. Others are process failures, like skipping approval gates in CI or using long-lived cloud credentials instead of short-lived federated ones.
Before changing anything, check the current setup for remote state, state locking, encryption, version pinning for providers and modules, plan review in CI, drift detection, blast radius per state, and credential lifetime.
Every experienced infrastructure engineer has a story about the day Terraform did exactly what it was told. That is the trap. Terraform is honest to a fault: it executes the plan it is given, whether or not that plan was the one anyone intended. In production, the gap between “the config looks right” and “the apply is safe” is where most incidents live. The mistakes below are not exotic edge cases. They are the recurring, ordinary failures that show up in postmortems across teams of every size. (Introduction to Terraform)
Local State and Missing Locking Corrupt Everything Downstream
The single most damaging habit is running Terraform with its default local state file. State is Terraform’s memory of what exists in the real world, and if it lives only on one engineer’s laptop, it is one lost hard drive away from gone. Worse, when two people apply from different machines, each has a different picture of reality, and both will confidently overwrite resources the other created.
State locking is the second half of the same problem. Even with a remote backend, concurrent applies can race each other and leave state corrupted or resources orphaned. Locking prevents two operations from touching the same state at once. Older setups relied on a separate locking mechanism alongside the storage bucket; newer backend versions have added native locking options, so check which behavior your backend version actually provides rather than assuming.
What actually works in practice
- Move state to a remote backend with encryption at rest and access controls before anything else.
- Enable locking and confirm it works by triggering two applies at once in a sandbox.
- Never commit `terraform.tfstate` or its backup files to Git. Add them to `.gitignore` on day one.
- Treat state as sensitive data, because it often contains secrets in plaintext.
A common mistake is assuming “we use a remote backend” is the same as “we are safe.” If the bucket is world-readable inside the organization, or if locking was never enabled, the risk is barely reduced.
Secrets End Up in Code, Variables, and State
Hardcoded credentials in `.tf` files are the classic finding, but the subtler version is secrets that leak through state. Anything Terraform creates and records, including generated passwords and keys, lands in state. That means state storage needs the same protection as a secrets manager, not the same protection as a config repo.
Practical guidance that holds up: keep secrets out of version control entirely, inject them at runtime from a managed secret store, and mark sensitive variables so they do not print in plan output. Marking a variable sensitive does not remove it from state, though. It only hides it from the console. Teams that treat the two as equivalent get surprised during audits.
Applying Without a Reviewed, Saved Plan
Running `terraform apply` directly, without first generating a plan file, review, is a quiet but constant source of unintended change. The workflow that works is boring and repeatable: run `terraform plan -out=tfplan`, review the diff carefully, then run `terraform apply tfplan`. The saved plan guarantees that what gets applied is exactly what was reviewed, not a fresh recalculation against a world that may have shifted in between. (Terraform (software)))
The review step matters as much as the file. Plans are long, and eyes glaze over. The habit worth building is scanning for the scary lines first: anything marked for destruction, anything replacing a database or a network, anything touching IAM. A plan that destroys twelve resources when you expected two is a stop sign, not a detail.
Drift and Manual Console Changes
Someone fixes a security group in the console during an incident. Nobody updates the code. Weeks later, the next apply quietly reverts the fix, and the incident returns. This is drift, and it is less a Terraform bug than an organizational one. (Ai mockup generator for product design)
The countermeasure is scheduled drift detection: run `terraform plan` on a schedule in CI and alert when the diff is non-empty. It converts silent divergence into a visible signal. The cultural half is harder. Make the console read-only for production where the cloud provider allows it, or at minimum require that any manual change be followed by a code change the same day.
One Giant State With an Enormous Blast Radius
A single root module managing the entire estate feels tidy. It is also a single point of failure. One bad apply can touch networking, databases, and DNS simultaneously, and a plan against it takes forever because Terraform refreshes everything. Splitting state by lifecycle and blast radius, so that networking, data stores, and application layers live in separate states, limits how much can break at once. It also speeds up plans dramatically, which makes review more likely to actually happen.
The trade-off is coordination. Cross-state references add complexity, and teams sometimes over-split into dozens of tiny states that are painful to reason about. The useful rule is to split where a failure should be contained, not where the folder structure looks prettier.
Unpinned Providers and Modules
Version constraints are the seatbelt of Terraform. Without them, a routine `terraform init` can pull a new major provider version with breaking changes, and the failure surfaces mid-apply in production. Pin providers and modules to explicit version ranges, commit the lock file, and upgrade deliberately in a branch with a full plan review.
A related mistake is treating upgrades as maintenance trivia. Provider major versions change resource behavior, and the changelog is required reading, not optional. Teams that schedule upgrades rather than react to them spend far less time firefighting.
Misusing `-target` and `destroy` in Production
The `-target` flag is designed for exceptional recovery situations, not routine work. Using it habitually leaves state and reality subtly out of sync, because targeted applies skip dependency updates elsewhere. It is a scalpel for emergencies, and the documentation warns against it as a general tool. (How to take a screenshot on a mac)
`terraform destroy` against a production state is the other landmine. Safeguards worth having: a separate production workspace or directory with its own credentials, deletion protection on critical resources at the cloud level, and a policy that destroy requires a second human. If the only thing standing between a typo and a deleted database is careful typing, that is not a safeguard.
CI/CD Without Gates or With Long-Lived Credentials
Automation that applies on every merge, with no approval step, converts a small code review mistake into an outage. Pipelines need a plan stage that posts the diff for review and an apply stage gated behind approval for production. The plan output is also useful documentation of what changed and when.
Credentials deserve equal scrutiny. Static cloud keys stored in CI secrets are long-lived and easy to exfiltrate. Short-lived federated credentials, where the pipeline exchanges its identity for temporary access, shrink that window dramatically. The setup takes an afternoon and removes an entire class of breach.
A Practical Checklist Before the Next Apply
| Area | Question to ask | Why it matters |
|---|---|---|
| State | Is it remote, encrypted, and locked? | Prevents loss and corruption |
| Secrets | Are any in code or readable in state? | State leaks are real leaks |
| Plan | Is a saved plan reviewed before apply? | Stops unintended change |
| Drift | Is plan running on a schedule? | Surfaces silent divergence |
| Blast radius | How much can one apply break? | Contains failures |
| Versions | Are providers and modules pinned? | Avoids surprise upgrades |
| Access | Are credentials short-lived? | Limits breach impact |
Walking through this table takes ten minutes and routinely finds two or three live problems. It is worth doing quarterly, not once.
Conclusion
Terraform rewards discipline and punishes shortcuts, but the shortcuts are all avoidable. Remote state with locking, secrets kept out of code and protected in state, saved plans that get reviewed, drift detection, sensible state splitting, pinned versions, careful use of `-target` and `destroy`, and gated pipelines with short-lived credentials. None of these are clever. Together they are the difference between infrastructure as code and infrastructure as a recurring incident. The teams that get this right are not the ones with the most sophisticated setups. They are the ones who made the boring safeguards non-negotiable.
Frequently Asked Questions
How often should Terraform plans be reviewed in production?
Every production apply should have a reviewed plan, with no exceptions for “small” changes. Small changes are exactly where destruction of a critical resource hides, because reviewers relax when the diff looks short. Automating the plan into pull requests makes review a habit rather than a negotiation.
Is it safe to store Terraform state in a Git repository?
No. State frequently contains secrets in plaintext and changes constantly, which makes Git a poor fit for both security and merge behavior. Use a remote backend with encryption and access controls, and keep state files out of version control entirely. (How to take a screenshot on windows)
Can multiple teams share a single Terraform state?
They can, but they usually should not. Shared state means shared locks, shared blast radius, and coordination on every apply. Splitting by team or lifecycle boundary reduces contention and contains failures, at the cost of managing cross-state references deliberately.
What is the difference between drift and a configuration error?
Drift is a mismatch between real infrastructure and state caused by changes made outside Terraform, such as console edits. A configuration error is a mistake in the code itself. Drift detection catches the first; code review and testing catch the second. Both need attention, and they are fixed differently.
Should `terraform apply` run automatically after merge?
For non-production environments, automatic apply is reasonable and saves time. For production, an approval gate between plan and apply is the safer default. The gate costs seconds and prevents the class of incident where a merged typo becomes an outage before anyone reads the diff.
How do teams detect manual changes made in the cloud console?
Scheduled `terraform plan` runs in CI are the simplest detector: any non-empty diff signals that reality and code have diverged. Pair that with cloud audit logs to identify who made the change, and a policy requiring the code to be updated the same day.
Apply the same rule to module sourcing: pulling modules directly from a branch rather than a tagged release means the code you reviewed is not necessarily the code that runs. Reference modules by version tag, and treat third-party modules with the same scrutiny as provider upgrades. ## Conclusion, Revisited None of the safeguards above require new tooling or a platform team. They require a decision, made once, that the boring path is the only path to production. Every incident traced back to Terraform tends to trace further back to a skipped plan review, a shared credential, or a console change nobody wrote down. Fix the process instead of treating only the symptom, and the tool stops being the story.
