Testing

Polo’s tests run against real infrastructure wherever they can: a real PostgreSQL with row-level security, a real Temporal dev server, real runner processes, the real Terraform CLI. Providers such as GitHub, Slack, AWS and MCP servers are replaced by local fixture servers. That choice sets the limit of what a green run means, which the last section spells out.

The suites

Suite	Where	Runner	Covers
Control plane acceptance	api/test/	node --test	Each module through its HTTP routes against a disposable PostgreSQL
Architecture	api/test/architecture/	node --test	Import and layer rules for every module
Runtime	tests/test_runtime_*.py, tests/test_supervisor_runtime.py	pytest	Workflows and activities against a Temporal dev server and the built control plane
Runners and tools	tests/test_*_runner.py, tests/test_aws_*.py, tests/test_gcp_*.py, tests/test_kubernetes_*.py, tests/test_sandbox.py, tests/test_skills.py	pytest	Each runner and the sandbox in isolation
Browser acceptance	tests/browser/*.py	Playwright scripts	The UI at desktop and phone widths against a real API
Actual-model journeys	tests/test_journeys_actual_model.py	pytest, opt-in	Two end-to-end journeys with a real model

Control plane tests (api/test/)

One folder per module, mirroring api/src/modules/, plus:

• api/test/acceptance/: cross-module flows such as operations-approvals.test.ts and operations-preauthorizations.test.ts.
• api/test/architecture/module-boundaries.test.ts: parses every source file and fails on a forbidden import. See Repository layout.
• api/test/support/: fixtures. database.ts is the important one.

createTestDatabase in api/test/support/database.ts gives each fixture a new database and a new non-owner application role, runs the real migrations, and drops both afterwards. Tests therefore exercise row-level security as production does: the application role cannot bypass the policies. Tests that need the database skip themselves when POLO_TEST_DATABASE_URL is not set.

Other files in support/ are local fixture servers for providers (slack-runtime-server.ts, jira-runtime-server.ts, mcp-runtime-server.ts, prometheus-runtime-server.ts and others) and runtime-server.ts, a built control plane that the Python tests start.

Python tests (tests/)

pyproject.toml sets testpaths = ["tests"] and puts the repository root on the Python path, so runtime and tools import directly.

• Runtime tests start the built Nest fixture server (api/dist/test/support/runtime-server.js) and their own Temporal dev server, then run real workflows. Most use a scripted model (Pydantic AI’s FunctionModel), so the test decides what the “model” answers and checks what the system does with it.
• Runner tests start a runner behind a loopback HTTP server (tests/runner_fixtures.py) and send it signed requests.
• Fixture modules such as tests/gitops_fixture.py, tests/registry_fixture.py, tests/cloudformation_fixture.py and tests/kind_cluster.py stand in for external systems.
• tests/fixtures/change-outcomes.json holds cases that both the TypeScript side and the browser side are tested against, so the server and the UI classify a change’s outcome the same way.

Browser tests (tests/browser/)

Each file is a standalone script, not a pytest test. It starts Polo on a free loopback port, creates a workspace, drives Chromium through Playwright, and removes what it created. tests/browser/README.md describes what each script covers.

Run them locally

You need Node 24 or newer, Python 3.12 with uv, Docker, and the Temporal CLI. Start a disposable PostgreSQL with pgvector and a Redis, and point the tests at them. The test role must be allowed to create databases and roles.

export POLO_TEST_DATABASE_URL=postgresql://user:pass@127.0.0.1:5432/polo_test
export POLO_TEST_REDIS_URL=redis://127.0.0.1:6379

Caution

Use a throwaway server. The suites create and drop databases, schemas and roles on it.

Control plane. npm test builds first, then runs every compiled test file one at a time.

cd api
npm ci
npm test

To run one file, build and call the Node test runner directly:

cd api
npm run build
node --test dist/test/operations/terraform.test.js

Python. Build the control plane first, because the runtime tests start it.

(cd api && npm ci && npm run build)
uv sync --locked
uv run --locked pytest -q -rs

-rs prints every skipped test with its reason. Run one file or one test in the usual pytest way:

uv run --locked pytest -q tests/test_runtime_supervisor.py

Browser.

uv run --frozen --with playwright==1.64.1 playwright install chromium
PYTHONPATH=. uv run --frozen --with playwright==1.64.1 python tests/browser/product_command_surfaces.py

Set POLO_BROWSER_EXECUTABLE to use an installed Chromium or Chrome instead.

Actual-model journeys. Opt-in. They need the API build, Terraform, the Temporal CLI and an OpenAI-compatible model endpoint:

POLO_TEST_REAL_MODEL_URL=http://127.0.0.1:8097/v1 POLO_TEST_REAL_MODEL_ID=<model id> \
uv run --locked pytest -s tests/test_journeys_actual_model.py

Tests that need extra inputs

Many end-to-end tests are gated on something a plain machine does not have. Without the input they skip and say why; they never pass silently.

Input	Enables
POLO_TEST_TOOLCHAIN_IMAGE	Sandbox and image-build tests that need the engineering toolchain image (build it from engineering.Dockerfile)
POLO_TEST_KIND_KUBECONFIG	Kubernetes and Helm tests on a disposable kind cluster
POLO_TEST_ARTIFACT_ENDPOINT, POLO_TEST_ARTIFACT_BUCKET, POLO_TEST_ARTIFACT_ACCESS_KEY, POLO_TEST_ARTIFACT_SECRET_KEY	Artifact tests against S3-compatible storage
POLO_TEST_REAL_MODEL_URL, POLO_TEST_REAL_MODEL_ID	Tests that call a real model
a terraform binary on PATH	Terraform runner tests

CI

.github/workflows/ci.yml runs on every push and pull request. Its jobs:

Job	What it does
api	Starts PostgreSQL (pgvector), Redis and a Temporal dev server; npm ci, npm run build, npm test
python	Builds the Nest fixture server, uv sync --locked, then uv run --locked pytest -q -rs; runs tests/test_harness.py separately with the optional deepseek extra
browser	Installs Chromium and runs a fixed list of tests/browser/ scripts
images	Builds api/Dockerfile, runtime.Dockerfile and tools.Dockerfile without pushing
manifests	Lints the Helm charts, validates the Compose files, runs ShellCheck on the entrypoints and actionlint on the workflows

Two things about CI are worth knowing before you trust a green run:

• CI builds no engineering toolchain image. The python job points POLO_TEST_DOCKER_SOCKET at a path that does not exist, so the sandbox tests report as skipped, and it deselects two tests that only check for the Docker CLI. Kind, Terraform-binary, MinIO, S3 and real-model tests also skip there. Run those locally where the inputs exist.
• Only some browser scripts run in CI. The browser job names each script it runs. A script in tests/browser/ that the job does not list is run by hand.

What a passing run proves

Be precise about this when you report a change as tested.

• It proves contract behaviour against local fixtures. Requests are signed and verified, approvals bind to digests, tenants are isolated, workflows survive replay.
• It does not prove compatibility with a live provider. Every test reroutes provider hosts to loopback fixtures. docs/CAPABILITIES.md states that nothing in the repository has been verified against a real external account.
• It does not prove model quality. Most end-to-end tests script the model. The recorded live runs used one local model route.

This is why the capability levels separate implemented, verified locally and verified with a connected provider. When you add a capability, pick its level by the strictest test that really ran. See Capabilities.

Where to go next

• Repository layout for the dependency rules the architecture test enforces.
• Run locally for starting the whole product on your machine.
• Capabilities for how test evidence maps to a maturity level.