Practical deployment of GCP Workload Identity Federation: Keyless Access, Short‑Lived Credentials, and Auditing

GCP Workload Identity Federation enables authorizing external identities (for example from a corporate identity provider, a CI/CD runner, or AWS) for GCP resources without permanent Google Service‑Account keys. The result is keyless access with short‑lived, automatically rotated tokens. In this article I explain practical prerequisites, architecture, common pitfalls, verification steps, concrete implementation steps, as well as audit and monitoring measures to keep operations secure and auditable.

Why Workload Identity Federation?

Workload Identity Federation (WIF) is a token‑exchange mechanism: an externally issued token (often an OIDC JWT; OIDC stands for OpenID Connect, an identity layer on OAuth 2.0) is exchanged via the Google Security Token Service (STS) for a temporary Google access token. Short‑lived credentials reduce the risk of persistent key leaks because tokens expire automatically. At the same time they simplify rotation and ensure that permissions are centrally controlled via IAM roles.

Essential components briefly explained

A quick overview of central terms:

• Workload Identity Pool: collection of trusted external identities. Technically a GCP resource that groups providers.
• Provider: defines an external identity source such as an OIDC issuer or AWS STS endpoint. Contains issuer URI and allowed audiences.
• Service Account: GCP‑internal identity with roles; external principals may impersonate this service account via impersonation.
• STS (Security Token Service): the Google endpoint for token exchange (subject_token → access_token).
• JWKS (JSON Web Key Set): public keys of the IdP used to verify JWTs. Rotations often cause outages if not handled correctly.

Prerequisites and initial architectural decision

Check organizational and technical prerequisites in advance: appropriate IAM rights to create pools and providers, a stable IdP with an available JWKS endpoint, NTP‑synchronized systems (time is central to token validity) and an audit project for long‑term log retention. Also decide whether to allow only specific audiences per provider and which claims should be used as attributes (e.g. repo‑ID, subject, email).

Concrete implementation guide

The order is important: Pool → Provider → Service Account → Binding → Test. Below you will find command‑based steps that you should run through in test projects.

1) Create Identity Pool

gcloud iam workload-identity-pools create my-pool 
  --project=PROJECT_ID 
  --location="global" 
  --display-name="My Identity Pool"

2) Create OIDC Provider

gcloud iam workload-identity-pools providers create-oidc my-oidc-provider 
  --project=PROJECT_ID 
  --location="global" 
  --workload-identity-pool="my-pool" 
  --display-name="AzureAD Provider" 
  --issuer-uri="https://login.microsoftonline.com/TENANT_ID/v2.0" 
  --allowed-audiences="api://my-app-client-id"

Why this matters: issuer URI and allowed audiences prevent arbitrary tokens from being accepted. Audiences are the target client IDs, i.e. the expected audience (aud) in the JWT.

3) Create Service Account and binding with attribute condition

gcloud iam service-accounts create my-app-sa 
  --project=PROJECT_ID 
  --display-name="Service Account for federated workloads"

PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format='value(projectNumber)')

gcloud iam service-accounts add-iam-policy-binding my-app-sa@PROJECT_ID.iam.gserviceaccount.com 
  --project=PROJECT_ID 
  --role=roles/iam.workloadIdentityUser 
  --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/my-pool/attribute.repository/my-app"

Best practice: Use precise attributes (here attribute.repository) or IAM conditions (conditions) to restrict access at a granular level.

4) Test the token exchange

curl -s -X POST https://sts.googleapis.com/v1/token 
  -H "Content-Type: application/x-www-form-urlencoded" 
  -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&
subject_token_type=urn:ietf:params:oauth:token-type:jwt&
subject_token=EXTERNAL_OIDC_TOKEN&
requested_token_type=urn:ietf:params:oauth:token-type:access_token&
&audience=//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/my-pool/providers/my-oidc-provider&
scope=https://www.googleapis.com/auth/cloud-platform"

If the exchange fails, check: a valid JWT, correct audience and issuer, JWKS reachability, and NTP skew.

Operationalize GCP Workload Identity Federation

The transition from proof-of-concept to production requires policies for lifecycle, testing, and observability. Operationalization includes automated tests, monitoring STS rates, JWKS monitoring, and regular IAM reviews.

Automated tests

Schedule CI jobs that regularly perform a full token exchange and execute a minimal API operation (e.g., read bucket metadata). Define thresholds for response times and error rates. Example of a simple BASH test script that tests the exchange and access:

#!/bin/bash
# exchange-and-test.sh
EXTERNAL_TOKEN="$1"
PROJECT_NUMBER="$2"
POOL="my-pool"
PROVIDER="my-oidc-provider"

RESPONSE=$(curl -s -X POST https://sts.googleapis.com/v1/token 
  -H "Content-Type: application/x-www-form-urlencoded" 
  -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&subject_token_type=urn:ietf:params:oauth:token-type:jwt&subject_token=${EXTERNAL_TOKEN}&requested_token_type=urn:ietf:params:oauth:token-type:access_token&audience=//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL}/providers/${PROVIDER}&scope=https://www.googleapis.com/auth/cloud-platform")

ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r .access_token)
if [ -z "$ACCESS_TOKEN" ] || [ "$ACCESS_TOKEN" == "null" ]; then
  echo "Token exchange failed: $RESPONSE" >&2
  exit 2
fi

# Test: list buckets (minimal permission vorausgesetzt)
curl -s -H "Authorization: Bearer ${ACCESS_TOKEN}" 
  "https://storage.googleapis.com/storage/v1/b?project=PROJECT_ID" | jq .

Monitoring, quotas and STS rate limits

Consider STS API quotas: high exchange rates (e.g., many small CI jobs) can hit limits. Monitor Google API error quotas and configure backoff/retry in clients. Additionally, set up alerts for unusual increases in impersonations or token exchanges.

# Beispiel: einfache Log‑Abfrage nach STS Exchanges
gcloud logging read 'protoPayload.methodName="google.iam.sts.v1.Sts.Exchange"' --project=PROJECT_ID --limit=50

JWKS‑Rotation und IdP‑Stabilität

JWKS rotations at the IdP are a frequent cause of outages: a new key pair is published while clients still cache old keys. Check the JWKS endpoint availability, the cache TTL of your clients, and coordinate rotations. A testing phase before production rollout prevents unexpected outages.

Audit‑ and forensic practices

Audit is particularly important with WIF because short‑lived tokens are not visible in inventories. Enable at least:

• Cloud Audit Logs: Admin Activity (changes), Data Access (API access, optional), System Event Logs.
• STS and impersonation events: these show token exchanges and who impersonated a service account.
• Export to BigQuery for long‑term analysis and forensic queries.

Example: BigQuery‑Query for Exchange‑Events

-- Suche nach Token Exchanges und Impersonation Events
SELECT
  protopayload_auditlog.authenticationInfo.principalEmail AS principal,
  timestamp,
  protopayload_auditlog.methodName AS method,
  resource.labels.project_id AS project_id
FROM `PROJECT_ID.logging_dataset.cloudaudit_googleapis_com_activity_*`
WHERE protopayload_auditlog.methodName LIKE "%workloadIdentityPools%"
  OR protopayload_auditlog.methodName LIKE "%Sts.Exchange%"
ORDER BY timestamp DESC
LIMIT 100;

Use these queries also for automated alerts (e.g., unexpected principals or high rates).

Specifically for Zammad operation

For Zammad installations that use GCS or Pub/Sub, the following points are recommended: limit roles to exactly the required actions (e.g., roles/storage.objectCreator instead of full admin), instrument upload flows so that every upload produces an audit‑log event, and run regular integrity checks of the object list against audit logs to detect lost or unauthorized uploads. Zammad configurations with external storage backends should have dedicated monitoring views so that upload errors, auth errors, or permission‑denied responses are noticed immediately.

Permission design: least privilege and role model

A stable WIF rollout often fails due to roles that are too coarse. Work with a role model that supports the following principles: Least Privilege (minimal privileges), Segregation (separate roles for read/write/audit), and Contextual Binding (access only with correct claims). Examples:

• roles/storage.objectViewer: for read access
• roles/storage.objectCreator: for upload jobs
• Custom role with exactly the required API methods when predefined roles are too broad

Review each role via an IAM rights review: which API methods are actually required? Remove anything that is not strictly necessary.

Migration of service account keys: stepwise replacement plan

Develop a stepped migration path instead of a hard cutover. Typical phases:

1. Inventory: Which services use service account keys? Use logging and secret‑management scans.
2. Parallel operation: implement WIF access in parallel to existing keys; use feature flags or config overrides.
3. Test run: production‑like tests with shadow traffic or test projects.
4. Stepwise Decommissioning: Rotate & Revoke Keys after successful test runs; delete Keys in Secret Manager.

Important: retain a proven recovery option (e.g., a short‑lived newly created, tightly scoped key) for the event of an unexpected outage, documented and auditable.

Create emergency key securely and delete it afterwards

Only for real emergencies: create the key, store it encrypted and delete it immediately after recovery. Example:

# Erzeuge temporären SA‑Key und speichere in Secret Manager
gcloud iam service-accounts keys create /tmp/temp-key.json 
  --iam-account=my-app-sa@PROJECT_ID.iam.gserviceaccount.com

# Upload in Secret Manager (verschlüsselt durch KMS)
gcloud secrets create emergency-sa-key --data-file=/tmp/temp-key.json --replication-policy="automatic"

# Nach Wiederherstellung: löschen
rm /tmp/temp-key.json
gcloud secrets delete emergency-sa-key --quiet

Practical troubleshooting checks

Systematic diagnostic steps when Token‑Exchange fails:

1. JWT‑validation: issuer (iss), audience (aud), exp/nbf. Use jwt-inspector or jq-based checks.
2. JWKS‑Reachability: curl > status, check key IDs (kid).
3. NTP‑skew: ntpq -p or chronyc tracking.
4. IAM‑Binding: Verify that the principalSet is referenced correctly (PROJECT_NUMBER, Pool, Provider, Attribute).
5. Cloud Audit Logs: search for Sts.Exchange entries and error messages.

# JWKS Check
curl -s https://login.microsoftonline.com/TENANT_ID/discovery/v2.0/keys | jq '.keys[] | {kid, kty, use}'

# NTP check (chrony)
chronyc tracking

# Check Sts.Exchange failures in logs
gcloud logging read 'protoPayload.methodName:"Sts.Exchange" AND severity>=ERROR' --project=PROJECT_ID --limit=50

Fallback strategy (technical and organizational)

Define clear, tested emergency procedures: temporary key creation and controlled secret management (Secret Manager), secondary providers in pools, and documented rollbacks to previous auth mechanisms. Each exception must be logged, time-limited and audited after completion. Organizationally, an incident owner should be appointed and a clear communication plan established that informs affected teams and describes revocation measures.

Concrete to-dos for production rollout

• Automated Exchange tests in CI with alerting.
• JWKS monitoring and coordination of rotations with IdP operators.
• Export all relevant logs (Audit, STS, Impersonation) to a secured BigQuery dataset.
• Regular IAM reviews and use of custom roles instead of broad roles/editor.
• Document and test the emergency key process.

Conclusion

GCP Workload Identity Federation is an efficient, secure method to allow heterogeneous workloads keyless access to GCP. Operational effort focuses on fine-grained attributes/conditions, JWKS stability, STS quota monitoring and a robust audit setup. For Zammad integrations and CI/CD runners outside of GCP, WIF provides a maintainable alternative to keys — provided that tests, monitoring and fallback plans are included from the outset. With a phased migration plan, clear roles and verifiable runbooks you reduce risk and create an auditable, repeatable process for keyless access.

Further concise overview: indispensable to-dos

• Test: automate Token Exchange from the target environment.
• Monitoring: monitor STS rates, impersonation events and JWKS availability.
• Audit: export logs, prepare BigQuery queries, set up alerts.
• Security: minimal IAM roles, audience/claim RESTrictions, regular IAM review.
• Fallback: Short-lived service account keys only for emergencies; rotate and delete them afterward.

Operations & Architecture Notes

In practice, a local token-exchange proxy is often deployed: it reduces direct STS calls, consolidates JWKS caches, implements backoff/circuit-breaker logic and provides centralized metrics. Ensure that caching strictly respects the token TTL—extended local tokens create uncontrollable revocation windows. Place the proxy in a secure network segment with restricted egress and tightly limit its privileges. Log the original principal ID as a dedicated trace field so audit queries can correlate exchanges with subsequent API accesses. Define SLAs for exchange availability and a tested, time-limited fallback (highly restricted, automatically rotating key). This enables integrations with custom enterprise software to be operated stably and with controlled risk.

Short-lived credentials are also important for this topic. The article puts these aspects into clear context and shows what matters in day-to-day operations.