Quick Answer: What are the most common mistakes when using gitlab ci in production?
The most common mistakes when using GitLab CI in production are hardcoded secrets, unprotected variables, legacy `only`/`except` rules, missing concurrency controls, and unpinned Docker images. Most of these problems live in a single file, `.gitlab-ci.yml`, and in the runner settings attached to your project, which means they can be fixed without rebuilding your whole delivery process. (GitLab CI/CD documentation)
Some mistakes are configuration errors, like running untrusted merge request code on shared runners. Others are design choices, such as cramming every step into one giant job and losing parallelism, or skipping protected environments before a production deploy. The difference matters because configuration issues tend to cause security incidents, while design issues quietly slow the team down.
Before changing anything, check your variable protection and masking, your use of `rules` versus legacy syntax, runner executor type and isolation, deployment approval gates, image version pinning, and pipeline timeout and retry settings.
A single misconfigured line in a pipeline file can expose a database password to every fork of a public repository. That is not a hypothetical fear; it is one of the most repeated failure patterns in continuous integration, and it usually surfaces at the worst possible moment, deep inside a release window. GitLab CI is a genuinely capable platform, but its flexibility means the defaults rarely protect a team that has not thought carefully about production. Below are the mistakes that show up again and again, why they hurt, and what actually works in practice.
Hardcoding Secrets in the Pipeline File
The first mistake is also the oldest. Someone needs a deploy token, so they paste it into `.gitlab-ci.yml`. It works, the pipeline goes green, and everyone moves on. Months later the repository is forked, mirrored, or accidentally made public, and the credential walks out the door with it.
The fix is straightforward: store credentials as CI/CD variables at the project or group level, and mark them appropriately. Masked variables keep values out of job logs, and protected variables only expose secrets to pipelines running on protected branches or tags. A variable that is neither protected nor masked is visible to pipelines triggered from merge requests, including requests from forks.
For teams with stricter requirements, external secret managers integrate directly with GitLab CI, so the pipeline fetches a short-lived credential at runtime instead of storing one at all. That approach costs more setup time but removes the long-lived secret entirely.
Leaving Variables Unprotected and Unmasked
This deserves its own section because it is the subtle version of the previous mistake. A team does the right thing and creates a variable, but forgets to tick the protected box. Now a merge request from an outside contributor can run a pipeline that reads the value. If that pipeline also prints environment information for debugging, the secret lands in a log that anyone with read access can open.
A practical habit that works: treat every new variable as guilty until proven safe. Ask three questions before saving it. Does this pipeline need it on unprotected branches? Should it appear in logs? Who can trigger a pipeline that uses it? If the answers are “no,” “no,” and “nobody outside the team,” protect and mask it.
Clinging to Legacy only/except Syntax
Older pipelines still use `only` and `except` to control when jobs run. GitLab has moved toward `rules`, which handles the same logic plus conditions on variables, file changes, and merge request events. Teams that stay on the legacy keywords end up writing convoluted combinations that nobody can read six months later.
The practical rule is simple. New pipelines should use `rules` exclusively. Existing pipelines should migrate job by job, starting with the deployment jobs where a wrong condition is most expensive. A deployment job that accidentally runs on a feature branch because of a sloppy `except` clause is exactly the kind of mistake that gets noticed at 2 a.m.
No Concurrency Controls on Deployments
Two pipelines racing to deploy to the same environment is a classic production incident. It happens when a merge to main triggers a deploy while a manual retry of the previous pipeline is still running. The result is a half-updated environment and a very confusing rollback.
GitLab offers two tools here. A resource group serializes jobs that share it, so only one deployment to a given environment runs at a time. The `interruptible` keyword lets GitLab cancel a redundant pipeline when a newer commit arrives, which saves runner minutes and prevents stale deploys. Neither is enabled by default, and plenty of teams never notice they exist until something breaks.
One Giant Job Instead of a Directed Graph
Some pipelines have a single job called `build-and-test-and-deploy`. It works, technically, but it takes forty minutes, gives no useful feedback, and cannot be retried in pieces. Splitting work into stages helps, but stages still run sequentially by default.
Using `needs` turns the pipeline into a directed acyclic graph, so independent jobs start as soon as their dependencies finish rather than waiting for an entire stage to clear. A test suite that does not depend on the lint job should not wait for lint. Teams that adopt this usually cut pipeline duration significantly without changing a single line of application code. For anyone comparing platforms while rethinking their setup, a look at the GitHub vs GitLab 2026 breakdown helps frame which pipeline model fits the team.
Unpinned Docker Images and Floating Tags
A job that runs inside `node:latest` or `python:latest` is a job that will break on a Tuesday for no apparent reason. The image maintainers push an update, a dependency shifts, and the build fails while the application code sits untouched.
Pinning to a specific digest or a precise version tag makes builds reproducible. The tradeoff is maintenance: someone has to bump the version deliberately. That is the point. A scheduled job that opens a merge request for image updates turns a surprise outage into a routine review.
Ignoring Runner Security and Isolation
Shared runners are convenient and, for public projects, genuinely risky. If a runner uses the shell executor on a host that also holds deploy keys or production credentials, then any job that runs on it inherits that access. Merge request pipelines from untrusted sources become an attack surface.
Docker or Kubernetes executors provide isolation, though even those need care with privileged mode and mounted volumes. The practical guidance is to separate runners by trust level: one set for untrusted merge request work, another for protected-branch deployments, with credentials available only to the second. This is the same principle behind the broader push for AI product photography generator for ecommerce tooling to run in sandboxed environments, where a compromised process cannot reach production assets.
Skipping Protected Environments and Approval Gates
Deploying to production from any branch, with no manual gate, is fast right up until it is catastrophic. Protected environments restrict who can deploy and which branches are allowed to. Deployment approvals add a human checkpoint for the environments that matter.
The mistake teams make is applying this to everything, which creates so much friction that people start bypassing the process. A better pattern is tiered: staging deploys automatically, production requires approval, and emergency hotfixes follow a documented shortcut that still leaves an audit trail.
No Timeouts or Retry Policy
A job that hangs consumes runner minutes indefinitely. Without a `timeout`, a stuck deployment can sit for hours, holding a resource group and blocking every subsequent deploy. Without `retry`, a transient network failure fails the whole pipeline and forces a manual restart.
Set a timeout that reflects reality. A unit test job does not need an hour. A deployment job needs enough time to finish a rolling update, and no more. Add retries only for failures that are genuinely transient, such as runner system failures, not for application errors that will simply fail again.
Treating Pipeline Config as Tribal Knowledge
When every project writes its own `.gitlab-ci.yml` from scratch, the organization accumulates dozens of slightly different approaches to the same problem. Nobody knows which one is current. New hires copy the nearest example, bugs and all.
The `include` keyword lets teams share configuration across projects, and CI/CD components package reusable pipeline logic with versioning. Centralizing the common pieces, then letting projects extend them, keeps security fixes from having to be applied in twenty places. It also makes onboarding faster, which is a benefit that rarely shows up in a cost model but always shows up in practice.
A Quick Comparison of Mistakes and Fixes
| Mistake | Why it hurts | Practical fix |
|---|---|---|
| Hardcoded secrets | Credential leaks on fork or mirror | CI/CD variables or external secret manager |
| Unprotected variables | Exposure to merge request pipelines | Protect and mask every sensitive variable |
| Legacy `only`/`except` | Unpredictable job triggers | Migrate to `rules` |
| No concurrency control | Overlapping deploys, broken environments | `resource_group` plus `interruptible` |
| Monolithic jobs | Slow feedback, no partial retries | Split jobs and use `needs` |
| Floating image tags | Non-reproducible, surprise breakage | Pin versions or digests |
| Weak runner isolation | Untrusted code reaches credentials | Separate runners by trust level |
| No approval gates | Accidental production deploys | Protected environments and approvals |
| Missing timeouts | Hung jobs burn runner minutes | Set `timeout` and targeted `retry` |
| No shared config | Drift and duplicated fixes | `include` and CI/CD components |
What Actually Works in Practice
The teams that run GitLab CI well in production tend to share a few habits. They review pipeline changes the way they review application code, because a pipeline change can deploy to production just as easily as a code change. They keep a small set of documented patterns and expect new projects to start from them. They test pipeline changes on a branch before merging.
They resist the urge to over-engineer. A pipeline with forty jobs and elaborate conditional logic is harder to debug than one with twelve clear jobs. Simplicity is a security feature; it also helps readability. They accept that some mistakes are inevitable, so they build in fast rollbacks rather than assuming nothing will ever go wrong.
One more practical note: pipeline debugging often involves capturing what happened on screen. Knowing how to take a screenshot on Windows or how to take a screenshot on a Mac sounds trivial until an incident review needs a clear record of a failed job log.
Conclusion
Most production problems with GitLab CI trace back to a small set of avoidable decisions: secrets in the wrong place, variables without protection, legacy syntax nobody wants to touch, missing concurrency controls, and runners that trust too much. None of these require a platform migration to fix. They require a deliberate review of the pipeline file and the project settings around it, ideally before the next release window rather than during it. Teams that treat their pipeline as production code, with the same review and versioning discipline, tend to spend far less time firefighting and far more time shipping.
Frequently Asked Questions
How often should a GitLab CI configuration be reviewed?
At minimum, whenever a deployment job changes, and on a regular cadence such as quarterly for the rest of the pipeline. Security-relevant settings like variable protection and runner assignment deserve a review whenever team membership changes.
Can a single shared runner handle both merge requests and production deploys?
It can, but it should not. Untrusted merge request code and production credentials on the same runner creates a direct path from a pull request to a live environment. Separate runners by trust level instead.
Is it safe to use latest tags for Docker images in CI jobs?
It is convenient and rarely safe. A floating tag means the build environment can change without any change to the repository, which makes failures hard to reproduce. Pin to a specific version or digest and update deliberately.
What is the difference between protected variables and masked variables?
Protected variables are only available to pipelines on protected branches and tags. Masked variables have their values hidden in job logs. They solve different problems, and sensitive credentials usually need both.
Do approval gates slow down delivery too much?
Only when applied indiscriminately. Gating production while letting staging deploy automatically keeps speed where it is safe and control where it matters. Emergency paths with audit trails prevent the gates from becoming bottlenecks people route around.
How can a team tell whether its pipeline is actually slow or just feels slow?
Measure job duration and queue time separately. Long queue times point to runner capacity, while long job durations point to missing parallelism or oversized jobs. The two problems have completely different fixes.
