Deploy with Helm

The chart in deploy/helm/polo deploys the API, the outbox relay, the scheduler and the runtime worker as Deployments, the database migration as a pre-install and pre-upgrade Job, a Service, an Ingress, optional watch processes and an optional autoscaler for the API. Read the process model, images and secrets in Deploy with Docker Compose first; they apply here unchanged.

Backing services

PostgreSQL with pgvector, Redis, Temporal and S3-compatible storage are external to the polo chart and referenced by URL. You have two options:

• Managed or existing services. Point the Secret and values at them.
• In-cluster services. The second chart, deploy/helm/polo-dependencies, runs PostgreSQL with pgvector, Redis and Temporal as single replicas on persistent volumes, with an optional daily database dump to S3 (backups.enabled). Install it as its own release before polo, because the migration hook needs PostgreSQL. It reads POLO_DATABASE_PASSWORD and POLO_REDIS_PASSWORD from one existing Secret (existingSecret, default polo-dependencies). With the release named polo-dependencies, the services are polo-dependencies-postgres:5432, polo-dependencies-redis:6379 and polo-dependencies-temporal:7233.

Note

The dependencies chart has no failover. An environment that outgrows single replicas should move to managed PostgreSQL, Redis and Temporal.

Install

1. Create the credentials Secret in the release namespace. The keys are the environment variable names the processes read.

   kubectl -n polo create secret generic polo-secrets \
     --from-literal=POLO_API_DATABASE_URL='postgresql://polo_application:...@postgres:5432/polo_product' \
     --from-literal=POLO_API_MIGRATION_DATABASE_URL='postgresql://polo_migration:...@postgres:5432/polo_product' \
     --from-literal=POLO_API_DISPATCH_DATABASE_URL='postgresql://polo_dispatch:...@postgres:5432/polo_product' \
     --from-literal=POLO_API_DATABASE_PASSWORD='...' --from-literal=POLO_API_DISPATCH_PASSWORD='...' \
     --from-literal=POLO_API_REDIS_URL='redis://:password@redis:6379/0' \
     --from-literal=POLO_API_WORKER_KEYS='{"worker":"...","scheduler":"..."}' \
     --from-literal=POLO_API_OPERATION_KEY='...' --from-literal=POLO_API_SSO_KEY='...' \
     --from-literal=POLO_API_ARTIFACT_ACCESS_KEY='...' --from-literal=POLO_API_ARTIFACT_SECRET_KEY='...' \
     --from-literal=POLO_RUNTIME_WORKER_TOKEN='...' --from-literal=POLO_SCHEDULER_WORKER_TOKEN='...'

   The ids in POLO_API_WORKER_KEYS must match workerIds.runtime and workerIds.scheduler (defaults worker and scheduler) and the workerId of each enabled watcher. With metrics.enabled: true the Secret also needs POLO_API_METRICS_TOKEN.

2. Create the model registry and model credentials Secrets. The registry is the same JSON as the local launcher’s models.json; the credentials Secret has one key per api_key_env named in the registry.

   kubectl -n polo create secret generic polo-models --from-file=models.json=./models.json
   kubectl -n polo create secret generic polo-model-credentials --from-literal=OPENAI_API_KEY='...'

3. Write a values file.

   publicUrl: https://<your-domain>
   image:
     api: { repository: <your-registry>/polo-api, tag: "<tag>" }
     runtime: { repository: <your-registry>/polo-runtime, tag: "<tag>" }
   temporal: { address: temporal-frontend.temporal.svc.cluster.local:7233, namespace: polo }
   artifacts: { bucket: polo-workspace-artifacts, region: eu-west-1 }   # endpoint: https://... for non-AWS stores
   hostedModelRoutes:
     - { id: openai-default, name: OpenAI default, model: gpt-4.1, shared: true, tenantIds: [] }
   api:
     trustedProxies: ["10.244.0.0/16"]   # where the Ingress controller connects from
   worker:
     models: { existingSecret: polo-models }
     modelCredentialsSecret: polo-model-credentials
     snapshots: { existingClaim: polo-snapshots }
   ingress: { className: nginx, tls: { secretName: polo-tls } }

4. Install or upgrade. The migrate Job runs first as a hook; the Deployments roll out after it succeeds.

   helm upgrade --install polo deploy/helm/polo -n polo -f values.yaml
   kubectl -n polo rollout status deployment/polo-api

To lint the chart locally, pass values, because publicUrl is required:

helm lint deploy/helm/polo -f deploy/helm/polo/ci/lint-values.yaml

Values you will most likely set

Value	Purpose
publicUrl	The HTTPS origin users open. Required.
secrets.existingSecret	The credentials Secret. Default polo-secrets.
api.trustedProxies	Addresses or CIDR blocks whose X-Forwarded-For the API believes. Required while the Ingress is enabled.
api.replicas, api.autoscaling.*, api.podDisruptionBudget.*	API scale. Default two replicas, autoscaling off.
temporal.address, temporal.namespace, temporal.taskQueue, temporal.tls, temporal.apiKey	The Temporal frontend. With apiKey: true the Secret also holds POLO_API_TEMPORAL_API_KEY and POLO_TEMPORAL_API_KEY.
artifacts.bucket, artifacts.region, artifacts.endpoint	Artifact storage. An empty endpoint means AWS S3 in region.
tls.caSecret	A Secret with a ca.crt the API trusts in addition to the system roots, for internal HTTPS endpoints with a private CA.
hostedModelRoutes	The model routes workspaces may choose; each mirrors a route of the worker’s registry.
worker.models.existingSecret, worker.modelCredentialsSecret	The model registry and its credentials.
worker.snapshots.existingClaim	A PersistentVolumeClaim for workspace snapshots; an emptyDir when empty.
scheduler.apiUrl, worker.apiUrl	The API origin for these processes. Defaults to publicUrl; must be HTTPS.
operationAllowedHosts, uptimeAllowedHosts	Optional comma-separated host allow-lists for registered endpoints and uptime probes.
watchers.<name>.enabled	Optional watch processes: calendar, gmailIntake, managedGmail, sourceFollowthrough.
metrics.enabled, metrics.serviceMonitor.enabled	Prometheus metrics; see Operate.
defaultRunner.*, publicRunnerImage	Polo-run runners; see Runners.

Chart behaviour worth knowing

• Trusted proxies. Every request reaches the API pods from the Ingress controller, so without api.trustedProxies the per-client rate limits see one address. List where the controller connects from: normally the pod CIDR, or the node addresses for a controller on hostNetwork. The controller must pass the real client address on. Any pod that can reach the API Service directly can then claim a client address, so on a shared cluster admit only the controller and Polo’s own pods with a NetworkPolicy.
• Probes. The API’s readiness probe is /readyz and its liveness probe is /healthz. A Temporal outage answers 200 with status degraded, so replicas stay in the Service.
• In-cluster traffic. The scheduler and worker reach the API through publicUrl, that is, through the Ingress with its public certificate. To use an internal HTTPS Service instead, set scheduler.apiUrl and worker.apiUrl, and tls.caSecret if that certificate comes from a private CA.
• Hardening. The API, relay and scheduler run with a read-only root filesystem and an emptyDir on /tmp. Every pod sets enableServiceLinks: false, because Kubernetes would otherwise inject a POLO_API_PORT variable that collides with the API’s own port setting.
• Development Secret. secrets.create: true renders the Secret from secrets.values as a pre-install hook. It is for development only and is not removed by helm uninstall.

The engineering sandbox

worker.engineering.enabled mounts the node’s Docker socket and workspaceRoot (a hostPath at the same path inside the pod) and adds dockerGid to the pod’s supplemental groups. The toolchain image must exist on that node’s Docker daemon.

Caution

This grants the worker pod root-equivalent access to its node. Dedicate nodes to the worker and pin it there with worker.nodeSelector. It is off by default.

Migrations and provisioning

Two idempotent commands run before every install or upgrade (and before every start in Compose):

• polo-api migrate runs the ordered migrations inside one transaction with the owner role from POLO_API_MIGRATION_DATABASE_URL and prints the applied names.
• polo-api provision creates or updates the restricted application role (POLO_API_DATABASE_ROLE, POLO_API_DATABASE_PASSWORD) and the dispatch role (POLO_API_DISPATCH_ROLE, POLO_API_DISPATCH_PASSWORD) with the grants the API and relay need.

The API runs as a non-owner role that cannot bypass row-level security, and schema synchronisation is disabled, so a schema change can only arrive through this step.

The role in POLO_API_MIGRATION_DATABASE_URL needs:

• Ownership of the database. migrate creates the polo_* schemas with their tables, policies and functions.
• CREATEROLE. provision creates the two runtime roles and grants them CONNECT. Without it the hook stops with “Runtime role provisioning failed”.
• The vector extension. pgvector is not a trusted extension, so unless the role is a superuser, a superuser (or your provider’s administrator role) creates it once before the first migrate.
• BYPASSRLS as well if the same role takes backups; see Backup and restore.

On an external PostgreSQL server, as a superuser:

CREATE ROLE polo_migration LOGIN CREATEROLE PASSWORD '...';
CREATE DATABASE polo_product OWNER polo_migration;
\c polo_product
CREATE EXTENSION IF NOT EXISTS vector;

On PostgreSQL 16 and later, leave the server setting createrole_self_grant empty. provision accepts the automatic ADMIN membership a non-superuser gets on roles it creates, and refuses any broader membership on a runtime role.

If the migrate Job fails, it is kept for inspection:

kubectl -n polo get jobs -l app.kubernetes.io/instance=polo

Upgrade order

1. Build and publish the new polo-api and polo-runtime images (and polo-tools and polo-engineering-tools if they changed). Read the release notes for new environment variables or Secret keys and add them first.
2. Back up PostgreSQL. Then run the migration: helm upgrade runs it as the pre-upgrade hook.
3. The API rolls out, then the relay, then the scheduler and watch processes. They all use the same image.
4. The runtime worker rolls out last. It drains in-flight activities during worker.terminationGracePeriodSeconds (120 seconds), and new workflow tasks wait on the Temporal queue until a worker polls again. Temporal keeps the state of running tasks.
5. Customer runners are upgraded independently by their owners from a new polo-tools image. Check the release notes for a contract change before upgrading the control plane ahead of the runners.

Caution

Migrations are forward-only. If a step fails after the migration ran, keep the new schema and fix forward when possible; the previous API image is not guaranteed to run against the new schema. Otherwise restore the backup from step 2 and redeploy the previous images.