Runners

A runner is a small service that carries out operations next to the systems it operates on: Terraform plans and applies, Helm releases, database operations, image builds, cloud reads and secret rotation. It runs inside your own network from the polo-tools image. The control plane never holds your cloud credentials. It sends signed operation requests; the runner verifies the signature, acts only within its configured scope, and returns a signed receipt.

For the workspace side (adding connections, the vault), see Runners and vault. For the code, see Tools and runners.

One runner per domain

The RUNNER environment variable selects what a container runs.

RUNNER	What it does	Env prefix
terraform	Terraform plans, stored encrypted, and one-shot applies of the approved plan	POLO_TF_RUNNER_
postgres	PostgreSQL backup, restore, query and migration	POLO_DB_RUNNER_
kubernetes	Deployment of exact Helm-rendered Kubernetes manifests	POLO_GITOPS_RUNNER_
containers	OCI image builds and approved publication to a registry	POLO_OCI_RUNNER_
aws	AWS inventory and Cost Explorer reads	POLO_AWS_RUNNER_
aws-secrets	Secret rotation; values never leave Secrets Manager	POLO_AWS_SECRETS_
cloudformation	Stack evidence, drift, change-set plans and exact change-set execution	POLO_CFN_RUNNER_
azure	Azure inventory, daily cost and approved virtual machine power changes	POLO_AZURE_RUNNER_
gcp	Compute Engine inventory, billing-export cost and approved instance power changes	POLO_GCP_RUNNER_

Each prefix takes CONFIG_FILE, STORAGE and CONFIG_KEY. The Terraform runner also takes PLAN_KEY and the Kubernetes runner BUNDLE_KEY. The exact configuration fields of each runner are in its module under tools/<domain>/ and in the Add connection dialog on the Connections page.

Two ways to connect a runner

Connect out to Polo (no public address)

This is the simple way. The container opens no port and needs no TLS certificate; it asks Polo for work over its own outgoing HTTPS connection. It can run on a laptop, a VM or a pod inside a private network, with outbound HTTPS to Polo as its only requirement.

1. In Polo, open Connections → Runners and add a runner. Polo shows a token once.

2. Start the tools image with Polo’s address and that token. With POLO_RUNNER_SETTINGS=polo the runner takes its settings from Polo, so it needs no RUNNER, no settings file and no signing key:

   docker run -d --restart unless-stopped --name polo-runner \
     -e POLO_URL=https://<your-domain> \
     -e POLO_RUNNER_TOKEN=<token> \
     -e POLO_RUNNER_SETTINGS=polo \
     -v polo-runner:/var/lib/polo \
     <your-registry>/polo-tools:<tag>

   The plungo command line does the same in one step, and --aws shares your ~/.aws with the runner read-only:

   plungo runner start --token <token> --image <your-registry>/polo-tools:<tag>
   plungo runner logs
   plungo runner stop

3. The runner shows as Connected under Runners. When you add a connection, pick the runner under Runs on instead of entering an address.

Details:

• Polo treats a runner as connected while it asked for work in the last 45 seconds. A request to a runner that is not connected fails at once as unreachable.
• Removing the runner on the Connections page stops its token.
• With POLO_RUNNER_SETTINGS=polo, Polo keeps the runner’s settings (its signing key and its targets), encrypted with the operation key. Each connection added on that runner becomes part of them. The runner stores them encrypted under POLO_RUNNER_STORAGE (default /var/lib/polo) and fetches them again when they change.
• For AWS on such a runner, a target without a role is read with the access the runner’s machine already has (an instance role, a mounted ~/.aws, environment credentials), and only when that access is in the target’s account.
• Without POLO_RUNNER_SETTINGS=polo, set RUNNER to one or more runners, comma-separated (for example RUNNER=aws,terraform), and give each its own encrypted settings file as described below. Such a runner refuses anything outside its file.

Caution

POLO_RUNNER_SETTINGS=polo trades control for simplicity: Polo, not a file on the runner, decides which targets the runner serves. Whoever administers the workspace in Polo controls what that runner reads with its machine’s access. A runner with its own settings file never asks Polo for settings, and Polo cannot widen what it does.

Polo calls the runner at a public HTTPS address

Run one container per runner, RUNNER=<domain>, listening on RUNNER_PORT.

1. Write and encrypt the configuration. Each runner reads a Fernet-encrypted configuration file (mode 0600, owned by uid 65532). It names the runner’s stacks, clusters, databases, registries or accounts, the separately scoped read and apply credentials, and the signing key. Generate a Fernet key with:

   python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'

2. Prepare storage. Storage must belong to uid 65532 with mode 0700 and no group or other access; otherwise the runner refuses to start (“Customer runner storage must be private”). The image ships /var/lib/polo/<runner> that way, and a new Docker named volume mounted there takes over that ownership. Set the runner’s STORAGE variable to that path, for example POLO_TF_RUNNER_STORAGE=/var/lib/polo/terraform. A bind mount or a volume elsewhere needs preparing first:

   sudo install -d -m 0700 -o 65532 -g 65532 <dir>

   Kubernetes fsGroup volumes add group permissions and are refused.

3. Serve HTTPS. Either set RUNNER_TLS_CERT and RUNNER_TLS_KEY (the runner serves TLS itself) or terminate TLS in a reverse proxy. The certificate must be valid for the hostname you register.

4. Start it. deploy/compose/runners.example.yml shows the Terraform and Kubernetes runners with TLS, configuration and storage mounts:

   cp deploy/compose/runners.example.yml runners.yml   # adjust paths, ports and which runners you need
   docker compose -f runners.yml up -d

5. Register it. On the Connections page choose Add connection, and enter the runner’s HTTPS address and its signing key.

The control plane rejects plain HTTP and any endpoint whose name resolves to a private, loopback or otherwise non-public address. Publish a runner inside a private network through a gateway with a public address, or use the connect-out mode instead. If you set POLO_API_OPERATION_ALLOWED_HOSTS (operationAllowedHosts in the chart), the runner’s hostname must be on that list.

Sandboxing requirements

• The Terraform and Kubernetes runners isolate their CLIs with bubblewrap, which needs unprivileged user namespaces. Docker’s default seccomp and AppArmor profiles deny those, so run these two with seccomp=unconfined and apparmor=unconfined (keep no-new-privileges and cap_drop: ALL), or supply a profile that allows clone, unshare and setns with namespace flags.
• The containers runner needs a Docker socket of its own for rootless BuildKit.
• The aws runners need outbound HTTPS only.

GitHub App for protected-branch checkout

The Terraform, Kubernetes and containers runners check out a target’s protected branch through a GitHub App with read access to repository contents. Set all three on the runner, or none; with none, checkout is refused.

Variable	Value
POLO_GITHUB_APP_ID	The GitHub App’s ID
POLO_GITHUB_APP_PRIVATE_KEY_B64 or POLO_GITHUB_APP_PRIVATE_KEY_FILE	Exactly one: the App’s unencrypted RSA private key in PEM form, base64-encoded or as a file path
POLO_GITHUB_APP_REPOSITORIES	JSON object of the repositories the runner may check out, for example {"owner/repo": {"installation_id": 12345678, "base": "main"}}

Each checkout requests a contents: read token for that one repository, confirms that the branch is protected and still at the approved commit, and extracts the archive of that commit.

The default runner

A workspace that installs no runner can use the default runner: Polo runs one small runner pod per workspace in its own cluster. It is off unless the chart’s defaultRunner.enabled is set, and it serves cloud APIs only. A system inside a private network still needs a runner installed there.

Set up once:

1. Publish the image built from tools.Dockerfile to a registry the cluster can pull, and set defaultRunner.image.
2. Create a namespace of its own and set defaultRunner.namespace. The chart gives the API’s service account the right to create and patch Secrets, volume claims and Deployments in that namespace, and nothing else.
3. Create the AWS role the runner pods act as and set defaultRunner.awsPrincipal. On EKS, bind it to the runner service account through defaultRunner.serviceAccount.annotations. The role needs only sts:AssumeRole.

What happens then:

• On a workspace’s first connection that chooses Polo’s default runner, the API creates a Secret with the runner’s token, a volume claim (storageSize, storageClass) and a Deployment, each named polo-runner-<workspace id>. The pod has no Service, mounts no cluster token, and runs as a non-root user with a read-only root filesystem.
• A request for a stopped runner starts it. The first request after a stop can time out while the pod starts; the next one succeeds. A runner nothing has needed for idleMinutes (default 15) is stopped; its volume stays.
• A workspace creates a read-only role in its own AWS account whose trust policy allows awsPrincipal and requires the workspace’s external ID. The connection form shows the policy.

publicRunnerImage is the image name Polo shows under Connections → Runners for a workspace that starts its own runner.

Trust and approval

• Signed requests. Every request carries an HMAC-SHA256 signature over the method, path, a timestamp (accepted within 60 seconds) and the body, computed with the connection’s signing key. The runner verifies it with the key in its own settings.
• Scope lives on the runner. A runner acts only on the stacks, clusters, databases or accounts its configuration names, with separate read and apply credentials.
• A runner token is not authority. In connect-out mode the token only lets the process collect its own workspace’s requests. Each request is still signed and verified.
• A person approves the exact change. An agent proposes an operation; it runs only after a person approves it or an administrator has pre-authorized that exact operation. The runner then applies only what was reviewed, for example the saved Terraform plan, and Polo checks the result with a separate read. See Approvals and guardrails and The operation contract.
• Receipts. Runners keep signed receipts and state in their storage directory. If a runner cannot confirm an outcome, the operation stays unknown until verified; check the runner’s receipt before retrying.

Rotate keys

• Signing key: generate a new key of at least 32 ASCII characters, update the runner’s encrypted configuration and restart it, then update the runner’s connection with the same key. Requests sent between those two steps are rejected with 401, so rotate while no operation is pending.
• CONFIG_KEY: re-encrypt the configuration file with the new key and restart.
• PLAN_KEY and BUNDLE_KEY: rotate only when no approved plan or bundle is pending. Stored plans encrypted with the old key become unreadable.

Upgrade runners

Runner owners upgrade independently from a new polo-tools image. Check the release notes for a request or receipt contract change before upgrading the control plane ahead of the runners.