Skip to main content

​
Kubernetes Deployment

The deployment checklist for the Rust workflow stack, and the gaps in the current manifests you need to know about before a production rollout.

​
Goal

A complete deployment means:
  • Temporal cluster available and healthy.
  • Temporal server pods isolated behind a Cloud SQL proxy sidecar.
  • Control-plane database migrations applied before app traffic.
  • engine-api, engine-worker, and dispatcher rolled out in a controlled order.
  • Per-brand connector secrets sourced from Vault and resolving correctly.
  • One brand onboarded in shadow for validation before live traffic.
Runtime components are the Temporal Helm release, engine-api, engine-worker, dispatcher, and the migrate job, in the stateset-temporal namespace.

​
Known gaps

These are current limitations of the shipped manifests, not steps to perform. Read them before planning a rollout — several change what you must do out of band.

​
Temporal schema is managed outside Helm

Helm manages the Temporal server Deployments, and the release uses a Cloud SQL proxy sidecar. Consequently the Helm chart schema jobs are disabled: the Temporal default and visibility databases must already exist, and schema setup and updates are handled outside the Helm install path.

​
Vault prerequisites must exist before brand onboarding

Required in stateset-temporal before onboarding any brand:
  • SecretStore/vault-backend
  • The external-secrets-vault-token Secret
  • One ExternalSecret per brand
  • next-temporal-v2 ServiceAccount permission to get namespace Secrets

​
Monitoring coverage is partial

The current overlay scrapes the API only. Worker and dispatcher metrics Services and ServiceMonitors are missing — so the dispatcher and connector metrics described in Operations are exposed by the pods but not collected.

​
Image versioning

Production should use immutable semver tags, release tags, or image digests. Combined with the worker rollback constraints, pinning matters more here than in a typical stateless service.

​
No environment overlays yet

The manifest layout is a single v2 stack — dev, staging, and prod overlays do not exist. Replica counts and some deployment assumptions are documented in more than one place; the manifests should become the source of truth once overlays land.

​
Phase 0 — cluster prerequisites

Confirm all of the following before starting:
  • Kubernetes cluster access and namespace creation permissions
  • Helm access
  • Ingress controller installed
  • cert-manager installed
  • external-dns installed, if hostname automation is required
  • External Secrets Operator installed
  • Prometheus Operator installed, if using ServiceMonitor and PrometheusRule
  • Managed PostgreSQL endpoints provisioned
  • Cloud SQL proxy credential secret available in stateset-temporal
  • Vault token secret available in stateset-temporal
  • Container registry pull access configured

​
Phase 1 — build and publish the image

Build the release image, push an immutable tag, record the digest, and update the manifests or overlay values to that exact tag or digest. For GitHub Actions, configure Workload Identity Federation secrets: GCP_WORKLOAD_IDENTITY_PROVIDER and GCP_ARTIFACT_REGISTRY_SERVICE_ACCOUNT.

​
Phase 2 — prepare secrets

Shared app secrets: CONTROL_PLANE_DATABASE_URL, CONTROL_PLANE_API_KEYS_JSON. Temporal Helm variables: TEMPORAL_CLOUDSQL_INSTANCE, TEMPORAL_CLOUDSQL_CREDENTIALS_SECRET, TEMPORAL_DB_NAME, TEMPORAL_DB_VISIBILITY_DB_NAME, TEMPORAL_DB_USER, TEMPORAL_DB_PASSWORD. Vault-backed brand connector secrets for the first rollout brand — Shopify, Gorgias, the LLM provider, the KB provider, and any other connector enabled in the brand manifest.

​
Phase 3 — install Temporal

Apply the namespace, ensure the Cloud SQL proxy credential secret exists in it, run the schema bootstrap script for a first install, then install or upgrade the Helm release. Wait for frontend, history, matching, and worker services to be healthy. Validate:
Confirm the default and visibility schemas were provisioned out of band.

​
Phase 4 — run the control-plane migration

Apply config, service account, and Vault SecretStore bootstrap resources, then apply only the migration job and wait for completion. Validate: control-plane tables exist, schema_migrations is current, and no failed migration job pods remain.

​
Phase 5 — roll out the app stack

In order:
  1. API Deployment, Service, PDB, Ingress
  2. Worker Deployment and PDB
  3. Dispatcher Deployment and PDB
Validate: /readyz on all three, the API can reach Temporal, the worker is polling the configured task queue, and the dispatcher health dashboard is healthy.

​
Phase 6 — monitoring

Apply ServiceMonitors and PrometheusRules. Confirm targets are scraped and alert rules load without errors.

​
Phase 7 — onboard the first brand in shadow

Import or validate the brand manifest, create the brand Vault entry, apply the brand ExternalSecret, then onboard with --activate. Validate: the brand’s routing mode is shadow, and connector secret resolution works.

​
Phase 8 — end-to-end smoke test

Start a workflow through the API and verify it appears in Temporal, the worker executes activities, dispatcher metrics are healthy, and there is no DLQ growth.

​
Phase 9 — promote to controlled traffic

Move from shadow to canary, watch the parity dashboard, and verify the mismatch rate stays within threshold. Only then consider live.
Phases 7 through 9 are the reason this is a staged rollout rather than a deploy. The app stack being healthy (Phase 5) tells you the engine runs; only the parity dashboard tells you it behaves the same as the system it replaces.