Automation: Set up a CI/CD pipeline with GitLab CI, Docker image scanning, and automated rollbacks

A reliable, automated delivery process reduces downtime and human error in the operation of digital enterprise solutions. In this article I explain in a practical way how to set up a CI/CD pipeline with GitLab CI, integrate Docker image scanning, and implement automated rollbacks. The goal is a maintainable operational path: Build → Test → Scan → Deploy → Observe → automatic rollback on critical failures. The focus keyword CI/CD-Pipeline mit GitLab CI is already placed in the introduction so the search intent is clear.

Why a CI/CD pipeline with GitLab CI?

GitLab CI is an integrated CI/CD system in GitLab that runs pipelines based on a .gitlab-ci.yml. Important for administrators: GitLab CI orchestrates runners (execution agents), has built-in registry integrations and can be connected to external tools (scanning, monitoring). A structured pipeline reduces deployment risk, automates security checks (image scanning) and enables safe deployments with the option to rollback.

Architecture overview and prerequisites

The typical architecture includes GitLab (repository + CI), GitLab Runners, a container registry (e.g. GitLab Registry or a private Harbor), an image scanning tool (e.g. Trivy, Clair or GitLab’s Container Scanning), an orchestration layer (commonly Kubernetes) and monitoring/alerting (e.g. Prometheus + Alertmanager). Prerequisites are:

• GitLab instance with runners and registry access.
• Authenticated registry credentials (tokens/CI_JOB_TOKEN or deploy keys).
• Target environment with a deploy API (kubectl + Kubeconfig or Helm).
• Monitoring with defined health and SLO metrics to support decision-making.
• Rollback mechanisms and playbooks documented for operations teams.

Terminology

CI/CD (Continuous Integration / Continuous Deployment) describes automated steps from code integration through delivery. Image scanning inspects container images for vulnerabilities and configuration issues. Rollback denotes reverting a deployment change when a release is faulty. SBOM (Software Bill of Materials) is an inventory of all components of an artifact and supports compliance and security analysis.

Pipeline design: stages, responsibilities, policies

A practical stage structure:

1. prepare: checkout, artifact preparation
2. build: image build and tagging
3. scan: image scanning (security/policy checks)
4. test: unit, integration and smoke tests
5. deploy: canary/primary deploy
6. verify: health checks, telemetry checks
7. promote / finalize: promotion after successful canary

Separation of duties is important: build and scan jobs should not run on the same runner as production deploy jobs, in order to minimize blast radius and privileges.

Image tagging strategy

Use immutable tags (e.g. Git commit SHA or semver plus build-id). Mutable tags like latest are problematic in production workflows because they impair reproducibility and rollback.

Example .gitlab-ci.yml: minimal but operationally safe

The following example shows the core parts: build, scan with Trivy, push and deploy into a Kubernetes cluster. Pay attention to defined secrets and a secure runner setup.

stages:
  - build
  - scan
  - test
  - deploy
  - verify

variables:
  IMAGE_REGISTRY: registry.example.local
  IMAGE_NAME: "$IMAGE_REGISTRY/myapp"
  KUBE_CONTEXT: production

build:
  stage: build
  image: docker:24
  services:
    - docker:dind
  script:
    - docker build -t "$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" .
    - docker push "$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"
  only:
    - main

scan:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy image --severity HIGH,CRITICAL --exit-code 1 --no-progress "$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"
  allow_failure: false

deploy_canary:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl --context="$KUBE_CONTEXT" set image deployment/myapp myapp="$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" --record
    - kubectl --context="$KUBE_CONTEXT" rollout status deployment/myapp --timeout=120s
  when: manual
  only:
    - main

verify_canary:
  stage: verify
  image: curlimages/curl:latest
  script:
    - /usr/local/bin/healthcheck.sh "$KUBE_CONTEXT" "myapp" || exit 1
  allow_failure: false

Why this works: Build produces an immutable image; Scan aborts the pipeline on critical vulnerabilities; Deploy uses kubectl to update the existing Deployment resource; Verify invokes a healthcheck that is based on runtime metrics or smoke tests. Failures in Scan/Verify prevent automatic promotion to production.

Image scanning: tools, policies and pitfalls

Common scanners are Trivy (fast, CLI-based), Clair (server), Grype or GitLab’s integrated Container Scanning (SAST/DAST). Scanners provide CVE lists, severity levels and fix versions. Choose the policy definition carefully:

• Blocking severity levels (e.g. CRITICAL/HIGH) for automatic abort decisions.
• Allow-listing for accepted but assessed vulnerabilities (with a review process).
• Regular database updates for the scanner (vulnerability feeds) – outdated DBs produce false negatives.

Typical pitfalls:

• Missing authentication against the registry: CI_JOB_TOKEN or deploy keys are absent.
• Image caching in runners masks problems: test scans against pulled images, not only local layers.
• Scanner versions vary in CVE coverage – document versions for audit.

Deployment strategies and automated rollbacks

Automated rollbacks require deployments to be reversible and observable. Common strategies:

• Blue/Green: a full parallel environment, traffic switch on success.
• Canary: partial traffic to the new release, observed metrics determine promotion.
• Rolling update with readiness probes: the Kubernetes default, but limited guarantees without monitoring.

Rollback mechanisms in Kubernetes

In Kubernetes the simplest rollback is kubectl rollout undo, which reactivates the previous ReplicaSet. The prerequisite is that ReplicaSets are retained (default: yes, unless explicitly cleaned up). For Helm deployments use helm rollback with stored releases.

# Rollback auf letzte Revision (kubectl)
kubectl --context=production rollout undo deployment/myapp

# Helm rollback auf Revision 2
helm --kube-context production rollback myapp 2

When that fails: rollbacks are ineffective when database migration steps cannot be rolled back in a compatible way or when configuration changes are not reversible. Plan database migrations as separate, controlled processes (e.g., migrations that are forward- and backward-compatible).

Implementing an automatic rollback trigger

An automatic rollback should not be executed blindly. Useful triggers are:

• Smoke tests (endpoint checks, auth flow, DB connectivity) fail.
• Error rate rises above defined thresholds (e.g., 5xx > X%).
• Latency or success rate falls below defined SLA values.

Technically, the trigger is implemented by a verification job in the pipeline or by external monitoring with a webhook. Example of a health check script that can be used in the verify-Stage:

#!/usr/bin/env bash
# healthcheck.sh: einfache Smoke-Checks für Canary
set -euo pipefail
KUBE_CONTEXT="$1"
DEPLOYMENT="$2"
NAMESPACE="default"
# Beispiel: 3 Versuche, 2 Sekunden Pause
for i in 1 2 3; do
  POD=$(kubectl --context="$KUBE_CONTEXT" -n "$NAMESPACE" get pods -l app="$DEPLOYMENT" -o jsonpath='{.items[0].metadata.name}')
  kubectl --context="$KUBE_CONTEXT" -n "$NAMESPACE" exec "$POD" -- /bin/sh -c 'curl -fsS --max-time 5 http://localhost:8080/healthz' && exit 0 || sleep 2
done
exit 1

If this script exits with code 1, an automatic rollback job should be executed in the pipeline or an alert should trigger an automated runbook action.

Practical pipeline extension: rollback job

Add a dedicated rollback job to the .gitlab-ci.yml that is only triggered when the Verify-Stage is missing. Important: rollback permissions should be RESTrictive (RBAC role with minimal privileges), and the rollback action should be idempotent.

rollback_on_verify_failure:
  stage: deploy
  image: bitnami/kubectl:latest
  when: on_failure
  script:
    - kubectl --context="$KUBE_CONTEXT" rollout undo deployment/myapp || true
  only:
    - main

Note: use when: on_failure deliberately — in complex pipelines this can produce unexpected side effects if multiple jobs fail. Test the behavior in a staging environment.

CI/CD pipeline with GitLab CI: operation, scaling and governance

Scaling and governance are as important in production as pipeline logic. Key operational topics:

• Runner scaling: use GitLab Runner autoscaling (e.g., Kubernetes-Executor or Docker Machine) to cover load spikes without manual intervention.
• Shared vs. specific runners: Shared Runner are convenient but increase the risk of resource contention. Use dedicated runners for privileged deploy jobs.
• Audit and compliance: enable audit logs in GitLab and retain scan reports and SBOMs for audits.

Example: excerpt from a GitLab Runner config.toml for a Kubernetes-Executor with autoscaling (simplified):

[[runners]]
  name = "k8s-runner"
  url = "https://gitlab.example.local/"
  token = ""
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "gitlab-runner"
    image = "alpine:3.18"
    idle_timeout = 1800
    poll_timeout = 180

Why this helps: the Kubernetes-Executor creates a pod per job and thus limits side effects from shared runner hosts. Pay attention to resource requests/limits so job pods do not exhaust node resources.

Secrets, RBAC und Least-Privilege

Secrets should never end up in the image. In GitLab use Protected Variables (masked, available only in protected branches). For deployments into the Kubernetes cluster, use ServiceAccounts with minimal RBAC rights — not cluster-admin tokens.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: production
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get","list","watch","update","patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployer-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: ci-deployer
  namespace: gitlab-runner
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

The Role/RoleBinding example above grants the ServiceAccount only deploy and rollback actions in the production namespace. This granularity reduces the risk from compromised CI tokens.

SBOM, Artefakt-Retention und Registry-Governance

SBOMs make dependencies and included libraries auditable. Trivy can generate SBOMs; attach them as artifacts to build jobs. Define an artifact and image retention strategy: overly aggressive garbage collection can prevent rollbacks, overly permissive policies can fill registry storage.

# Generate Trivy SBOM
trivy image --format cyclonedx --output sbom.cdx.json registry.example.local/myapp:$CI_COMMIT_SHORT_SHA

Recommendation: Retain at least the last N images per service (e.g. N=10) and SBOMs for statutory retention periods where compliance requires it.

Observability und Alert-to-CI-Integration

Automated decisions are based on metrics. Define clear alert rules in Prometheus and connect Alertmanager to a webhook that calls a GitLab trigger or creates a ticket. Example of a simple Prometheus alert rule (simplified example):

groups:
- name: app.rules
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{job="myapp",status=~"5.."}[2m]) > 0.05
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High error rate in myapp"
      runbook: "https://wiki.example.local/runbooks/myapp-rollback"

Alertmanager can send a webhook to a CI trigger endpoint when fired. Ensure trigger endpoints are secured and only authorized alerts can trigger actions.

Garbage Collection, Caching und Performance-Hinweise

Scanner performance depends heavily on network availability and local cache strategy. Avoid having runners repeatedly download large images by using a local registry cache or proxy. Also, use Docker-in-Docker only when runner isolation is not achievable otherwise; Kaniko, BuildKit or Podman are often safer alternatives for the Kubernetes executor.

Test- und Validierungsstrategie vor produktivem Rollout

Test every step:

• Validate runner and registry access using dummy images.
• Verify scanner DB updates and exit codes (Trivy –version, trivy db update).
• Perform repeated deploy/rollback cycles in staging; check ReplicaSet and Helm release histories.
• Simulate health-check failures to verify automated rollbacks.

# Trivy DB update (important before scans)
trivy db update

Typische Fehlerfälle und Troubleshooting

Error: Scan jobs take very long or timeout. Causes: scanner DB outdated, network proxy blocks access to vulnerability feeds, large image layer cache. Checks: verify scanner version, test network access, validate image for local scanner execution.

Error: Rollback fails because ReplicaSets were deleted. Cause: garbage collection / cluster cleanup. Mitigation: configure cluster policies to retain previous ReplicaSets/releases for a defined period.

Error: Database incompatibility during rollback. Cause: migrations that cannot be reverted. Resolution: two-step migrations (forward compatible), use feature flags, plan separate database rollback strategies.

Runbook: Quick step-by-step actions during an incident

1. Assess alert: metrics, logs, deployment history.
2. Isolate: immediately switch back canary traffic or revert the service to previous replicas.
3. Perform rollback (kubectl rollout undo or helm rollback).
4. Post-mortem: analyze root cause, scanner results, test coverage, metrics.
5. Lessons learned: adjust pipeline/tests/probes, deliver a patch.

Conclusion: Operational maturity rather than mere automation

Automated CI/CD pipelines with GitLab CI, integrated Docker-Image-Scanning and clearly defined rollback processes substantially reduce risk, but require discipline: immutable image tags, clean scanner policies, observability-based decisions and database strategies that support rollback. Test rollbacks regularly in staging, document runbooks and keep access rights RESTrictive. Only then does automation in operations become genuinely resilient instead of a new source of failures.

Next steps: Start with a small proof-of-concept: Build → Trivy-Scan → Canary-Deploy → Verify → Rollback, and extend the pipeline incrementally with monitoring and compliance checks. Schedule regular exercises (chaos tests, rollback drills) and audits of runner/registry policies.

Docker Image Scanning and Canary Deployment are also important for this topic. The article places these aspects into context and shows what matters in day-to-day operations.