Secret Rotation Procedures

All secrets in Gatelithix Gateway are stored in Google Cloud Secret Manager. Rotation scripts are located in the scripts/ directory and follow a consistent pattern: generate or accept a new value, create a new Secret Manager version, and keep the previous version ENABLED for rollback safety.

Secret Inventory

Auto-Generated Secrets

These secrets are generated by the rotation script itself. No external input is needed.

HMAC Signing Key

Used for request signing and webhook signature verification.

Rotation command:

scripts/rotate-hmac-key.sh --project gatelithix-core

Pre-rotation checklist:

• Verify current gateway health: curl -s https://api.gatelithix.com/health
• Confirm no active deploys in progress
• Note current secret version for rollback reference

What the script does:

1. Generates a new 256-bit random key
2. Creates a new Secret Manager version for hmac-signing-key
3. Keeps the previous version ENABLED (rollback safety)

Post-rotation verification:

# Verify the new version was created
gcloud secrets versions list hmac-signing-key --project gatelithix-core --limit 3
 
# Restart gateway to pick up the new key
gcloud run services update api-gateway \
  --region us-central1 --project gatelithix-core \
  --update-env-vars="RESTART_TRIGGER=$(date +%s)"
 
# Verify health after restart
curl -s https://api.gatelithix.com/health/ready

Rollback:

# If the new key causes issues, the gateway will automatically use "latest"
# Disable the broken version and the previous (still ENABLED) version becomes latest
gcloud secrets versions disable VERSION_NUMBER \
  --secret hmac-signing-key --project gatelithix-core

Database Passwords

Used for Cloud SQL authentication (core and PCI databases).

Rotation command:

# Core database
scripts/rotate-db-password.sh --project gatelithix-core --secret core-db-password
 
# PCI database
scripts/rotate-db-password.sh --project gatelithix-pci --secret pci-db-password

Pre-rotation checklist:

• Verify current database connectivity
• Confirm no active migrations running
• Note current secret version for rollback reference

What the script does:

1. Generates a new random password
2. Updates the Cloud SQL user password
3. Creates a new Secret Manager version
4. Keeps the previous version ENABLED (rollback safety)

Post-rotation verification:

# Verify the new version was created
gcloud secrets versions list core-db-password --project gatelithix-core --limit 3
 
# Restart the affected service to pick up the new password
gcloud run services update api-gateway \
  --region us-central1 --project gatelithix-core \
  --update-env-vars="RESTART_TRIGGER=$(date +%s)"
 
# Verify health
curl -s https://api.gatelithix.com/health/ready

Rollback:

# Disable the new version, then update the Cloud SQL user password back
gcloud secrets versions disable VERSION_NUMBER \
  --secret core-db-password --project gatelithix-core
# Re-run the rotation script to set the DB password to match the now-latest secret version

Redis Auth String

Used for Memorystore Redis authentication.

Rotation command:

scripts/rotate-redis-auth.sh --project gatelithix-core

Pre-rotation checklist:

• Verify current Redis connectivity
• Confirm no rate limiting incidents in progress
• Note current secret version for rollback reference

What the script does:

1. Generates a new auth string
2. Updates the Redis instance auth string
3. Creates a new Secret Manager version for redis-auth-string
4. Keeps the previous version ENABLED (rollback safety)

Post-rotation verification:

# Verify the new version was created
gcloud secrets versions list redis-auth-string --project gatelithix-core --limit 3
 
# Restart gateway to pick up the new auth string
gcloud run services update api-gateway \
  --region us-central1 --project gatelithix-core \
  --update-env-vars="RESTART_TRIGGER=$(date +%s)"
 
# Verify Redis connectivity via health check
curl -s https://api.gatelithix.com/health/ready

External Secrets (Manual Input)

These secrets come from external dashboards (PSP portals, Auth0). The rotation script prompts for the new value and updates Secret Manager.

Auth0 Client Secret

Used for Auth0 M2M token validation.

Rotation command:

scripts/rotate-auth0-secret.sh --project gatelithix-core

Steps:

1. Log into Auth0 Dashboard  > Applications > Gatelithix Gateway API
2. Rotate the client secret in Auth0
3. Copy the new secret
4. Run the script — it will prompt for the new secret value
5. The script creates a new Secret Manager version

Post-rotation verification:

# Restart gateway
gcloud run services update api-gateway \
  --region us-central1 --project gatelithix-core \
  --update-env-vars="RESTART_TRIGGER=$(date +%s)"
 
# Test authentication
curl -s -H "Authorization: Bearer $TEST_TOKEN" https://api.gatelithix.com/v1/merchants | head -1

Connector API Keys (Stripe, NMI, FluidPay)

Used for authenticating with payment service providers.

Rotation command:

# Stripe API key
scripts/rotate-connector-key.sh --project gatelithix-core --secret stripe-api-key
 
# Stripe webhook secret
scripts/rotate-connector-key.sh --project gatelithix-core --secret stripe-webhook-secret
 
# NMI API key
scripts/rotate-connector-key.sh --project gatelithix-core --secret nmi-api-key
 
# NMI webhook signing key
scripts/rotate-connector-key.sh --project gatelithix-core --secret nmi-webhook-signing-key
 
# FluidPay API key
scripts/rotate-connector-key.sh --project gatelithix-core --secret fluidpay-api-key

Steps:

1. Log into the PSP dashboard:
   • Stripe: Dashboard  > Developers > API keys
   • NMI: Merchant portal > API Settings
   • FluidPay: Admin portal > API Keys
2. Generate or roll the API key in the PSP dashboard
3. Run the rotation script — it will prompt for the new key value
4. The script creates a new Secret Manager version

Post-rotation verification:

# Restart the affected connector service
gcloud run services update stripe-connector \
  --region us-central1 --project gatelithix-core \
  --update-env-vars="RESTART_TRIGGER=$(date +%s)"
 
# Test connector health
gatelithix connector-health

Rollback: For connector keys, rollback requires re-activating the previous key in the PSP dashboard (if the PSP supports it). Disabling the new Secret Manager version alone is insufficient — the PSP must also recognize the old key.

General Rollback Procedure

All rotation scripts keep the previous Secret Manager version ENABLED. To roll back:

1. Identify the problematic version number
2. Disable it: gcloud secrets versions disable VERSION --secret SECRET_ID --project PROJECT
3. The latest alias now points to the previous (still enabled) version
4. Restart the affected Cloud Run service to pick up the reverted secret
5. For database and Redis secrets, also revert the underlying credential to match