Operate Polo

This page is about watching Polo itself: the API, the outbox relay, the scheduler, the watch processes and the runtime worker, and their dependencies (PostgreSQL, Redis, Temporal and S3-compatible storage). It does not cover the observability sources agents query on your behalf; for those see Connections.

Probes

Process	Path	Meaning
API	GET /healthz	Liveness only. It never fails on a dependency outage, so a database incident does not restart healthy replicas.
API	GET /readyz	200 with "status":"ready" and checks for database, redis and temporal. 200 with "status":"degraded" when only Temporal is down. 503 with Retry-After: 5 when PostgreSQL or Redis is down.
API	GET /metrics	Prometheus text. Requires Authorization: Bearer $POLO_API_METRICS_TOKEN; 404 when the token is unset, 401 when it is wrong.
Scheduler	GET /healthz, /readyz, /metrics on its own listener (default 127.0.0.1:9464)	The listener starts only when a metrics token is set. Ready while the last claims call reached the API within 60 seconds.

The Helm chart wires /healthz to the liveness probe and /readyz to the readiness probe. In Compose, the api healthcheck uses /readyz, and the scheduler and worker wait on it.

Enable metrics

Variable	Purpose
POLO_API_METRICS_TOKEN	32 or more printable characters. Enables GET /metrics on the API; also the scheduler’s fallback token.
POLO_SCHEDULER_METRICS_TOKEN	A scheduler-specific token, when it should differ.
POLO_SCHEDULER_METRICS_HOST, POLO_SCHEDULER_METRICS_PORT	Bind address of the scheduler listener (default 127.0.0.1 and 9464).

• Compose: set POLO_METRICS_TOKEN in .env. It serves both processes; the scheduler’s port is published as POLO_SCHEDULER_METRICS_PORT.
• Helm: set metrics.enabled: true and add POLO_API_METRICS_TOKEN to the Secret. The chart also creates the <fullname>-scheduler Service.

Scrape configuration

The API serves plain HTTP on 8088, so scrape every replica directly on the host or the private network, not through the public origin. Use the job names polo-api and polo-scheduler; the alert rules select on them.

Compose, with Prometheus on the Docker host:

scrape_configs:
  - job_name: polo-api
    authorization: { type: Bearer, credentials_file: /etc/prometheus/polo-metrics-token }
    static_configs: [{ targets: ['127.0.0.1:8088'] }]
  - job_name: polo-scheduler
    authorization: { type: Bearer, credentials_file: /etc/prometheus/polo-metrics-token }
    static_configs: [{ targets: ['127.0.0.1:9464'] }]
rule_files: ['/etc/prometheus/rules/polo.yml']   # scripts/ops/prometheus-rules.yml

Kubernetes with the Prometheus Operator: set metrics.serviceMonitor.enabled, and metrics.serviceMonitor.labels to whatever your Prometheus selects monitors by. The chart’s ServiceMonitors carry the token reference and the job names. Without the operator, both Services carry prometheus.io/scrape, /port and /path annotations; the scrape job still has to present the bearer token.

Metrics

Everything Polo emits is prefixed polo_. Node process defaults (process_*, nodejs_*) are on. Request metrics are labelled with the route pattern (/api/tasks/:id), never the concrete URL, so label cardinality stays bounded.

Metric	What it tells you
polo_http_requests_total, polo_http_request_duration_seconds	Traffic and latency by method, route and status
polo_ready, polo_dependency_up	The last readiness result, and each dependency (database, redis, temporal)
polo_db_pool_connections, polo_db_pool_idle_connections, polo_db_pool_waiting_requests, polo_db_pool_max_connections	Database pool pressure
polo_tasks	Tasks by status (queued, running, succeeded, failed, canceled)
polo_outbox_pending_events, polo_outbox_oldest_pending_age_seconds	The relay’s backlog
polo_operations_awaiting_approval, polo_operations_oldest_awaiting_approval_age_seconds	Proposed changes a person can still decide
polo_operations_undecidable_requests	Requests whose task or agent stopped; they never execute
polo_scheduler_heartbeat_age_seconds	Scheduler heartbeat per cell_id
polo_worker_command_rejections_total	Rejected worker commands by route and status
polo_scheduler_claims_total, polo_scheduler_completions_total, polo_scheduler_probe_failures_total, polo_scheduler_loop_failures_total	Scheduler throughput and failures (scheduler process)
polo_telemetry_collection_duration_seconds, polo_telemetry_collection_failures_total, polo_telemetry_tenants_skipped	Health of the collector itself

The backlog gauges are collected every 30 seconds with a few short statements per workspace, capped at 500 workspaces per cycle. A failed cycle leaves the previous values in place.

Dashboards to build

One Grafana board with three rows covers it.

Traffic and latency

sum by (route) (rate(polo_http_requests_total{job="polo-api"}[5m]))
sum(rate(polo_http_requests_total{job="polo-api",status=~"5.."}[5m])) / sum(rate(polo_http_requests_total{job="polo-api"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(polo_http_request_duration_seconds_bucket{job="polo-api",route!~"/metrics|/static/.*|unmatched"}[5m])))

Control plane

sum by (status) (polo_tasks)
max(polo_outbox_oldest_pending_age_seconds)
max(polo_operations_oldest_awaiting_approval_age_seconds) / 3600
min by (cell_id) (polo_scheduler_heartbeat_age_seconds)
sum by (route, status) (rate(polo_worker_command_rejections_total[5m]))
min by (instance) (polo_ready)

Capacity

polo_db_pool_waiting_requests
rate(process_cpu_seconds_total[5m])
process_resident_memory_bytes
nodejs_eventloop_lag_p99_seconds

Alerts

scripts/ops/prometheus-rules.yml holds 16 rules. Load the file into Prometheus as shown above.

Alert	Condition	Severity
PoloApiDown	up{job="polo-api"} == 0 for 1m	page
PoloApiNotReady	polo_ready == 0 for 2m	page
PoloDependencyDown	polo_dependency_up == 0 for 2m	page
PoloApiHigh5xxRate	5xx ratio above 2% for 5m	page
PoloApiLatencyP95High	p95 above 1 s for 10m	ticket
PoloDbPoolSaturated	polo_db_pool_waiting_requests > 0 for 5m	ticket
PoloTelemetryCollectionFailing	more than 3 failed cycles in 15m	ticket
PoloSchedulerHeartbeatStale	heartbeat age above 120 s for 2m	page
PoloSchedulerLoopFailing	more than 5 loop failures in 10m	ticket
PoloSchedulerProbeTransportFailures	transport failures above 0.2/s for 15m	ticket
PoloApprovalQueueAged	oldest awaiting approval above 4 h for 15m	ticket
PoloDeliveryBacklogStalled	oldest undelivered event above 5 min for 5m	page
PoloDeliveryBacklogGrowing	more than 100 undelivered events for 10m	ticket
PoloTasksQueuedNotStarting	queued tasks, empty outbox, no successful work commands for 10m	page
PoloWorkerRejectionSpike	more than 0.5 rejections/s for 10m	ticket
PoloWorkerUnauthorized	401 rejections above 0.05/s for 5m	page

Validate the rules after editing them:

docker run --rm -v "$PWD/scripts/ops:/rules:ro" --entrypoint promtool prom/prometheus:v3.5.0 check rules /rules/prometheus-rules.yml

Load test

scripts/ops/loadtest.mjs needs Node 24 or newer and has no dependencies. It signs in once and runs closed-loop workers over the reads the Home, Tasks, Incidents, Approvals and Agents pages issue, creating one task every --create-every requests. It reports throughput and p50, p95, p99 and max latency per endpoint, and exits with code 1 when any request failed.

node scripts/ops/loadtest.mjs --target https://<staging-origin> \
  --email <admin email> --password <password> --polo <agent id> \
  --concurrency 20 --duration 60 --create-every 25

POLO_LOADTEST_PASSWORD can carry the password instead of the flag. On a disposable target, --bootstrap creates a throwaway workspace, administrator and agent instead of signing in.

Caution

Never point the load test at production or at the local product on port 8088. It creates real tasks.

Measure on your own infrastructure before making capacity claims. The one metric to watch during a run is polo_db_pool_waiting_requests: when it is above zero while connections equal the pool maximum, the pool is the bottleneck.

On-call runbook

API down

Alerts: PoloApiDown, PoloApiNotReady, PoloDependencyDown.

1. Request /healthz, then /readyz. If liveness is up and readiness is 503, read checks:
   • database: false — PostgreSQL is unreachable, out of connections or in recovery. Restore the database; the API reconnects without a restart. If connections are exhausted, keep POLO_API_POOL_SIZE times the replica count below the server’s max_connections.
   • redis: false — sign-in and other rate-limited routes return 503 until Redis returns; the rest of the API keeps serving.
   • temporal: false — /readyz still answers 200 as degraded. New tasks stay queued until Temporal returns. Check the Temporal frontend, TLS and API key settings, and network policy.
2. If liveness is down, the process crashed or cannot start. Read its log for the startup error: a settings validation failure, a database role that is an owner or can bypass row-level security, or Redis unreachable at start. Fix the setting and restart.
3. If every replica is down after a deploy, roll back to the previous image. Migrations are forward-only, so a rollback must not require a schema revert.
4. Confirm polo_ready is 1 on every instance afterwards.

API errors and latency

Alerts: PoloApiHigh5xxRate, PoloApiLatencyP95High, PoloDbPoolSaturated, PoloTelemetryCollectionFailing.

1. Find the routes: topk(5, sum by (route) (rate(polo_http_requests_total{status=~"5.."}[5m]))). Each 5xx is logged as request.failed with a request id, route, error class and SQLSTATE code, never SQL text or bodies.
2. Pool pressure: raise POLO_API_POOL_SIZE within the database’s connection budget, or add a replica. If requests wait while the database is idle, look for long transactions.
3. Latency without pool pressure: check nodejs_eventloop_lag_p99_seconds (scale out), memory growth, and the database’s slow query log.
4. PoloTelemetryCollectionFailing means the backlog gauges are stale, not the product. Look for telemetry.collection_failed in the API log.

Scheduler stale

Alert: PoloSchedulerHeartbeatStale. The Automation page shows the scheduler offline, and schedules and uptime monitors stop producing tasks.

1. Check the process: its /healthz, its log, and the rate of polo_scheduler_loop_failures_total.
2. Check authentication: 401s on /api/internal/v1/automation/claims mean POLO_SCHEDULER_WORKER_ID or POLO_SCHEDULER_WORKER_TOKEN does not match the API’s POLO_API_WORKER_KEYS entry or POLO_API_AUTOMATION_WORKER_ID.
3. Check reachability: POLO_SCHEDULER_API_URL must be the API origin (HTTPS, or loopback HTTP).
4. If the heartbeat ages while claims still increase, the API and scheduler disagree on POLO_API_CELL_ID.
5. After a restart the heartbeat refreshes on the first claim. Monitors catch up on their next interval; missed schedule runs are advanced, not replayed.

Worker stopped

Alerts: PoloDeliveryBacklogStalled, PoloDeliveryBacklogGrowing, PoloTasksQueuedNotStarting, PoloWorkerRejectionSpike. Tasks stay queued or running without progress.

1. Find the stage. A growing polo_outbox_pending_events means the relay is down or Temporal refuses workflow starts. An empty backlog with tasks queued means no runtime worker polls the task queue.

2. Relay: check its process and log, POLO_API_DISPATCH_DATABASE_URL, and the Temporal address and task queue. Restarting the relay drains the backlog by itself.

3. Temporal: temporal task-queue describe --task-queue <queue> must list pollers from the runtime worker.

4. Worker: check its log for model or credential errors, then restart it.

   docker compose restart worker
   kubectl rollout restart deployment/<fullname>-worker

   Runs the worker had claimed become claimable again when their claim expires (POLO_API_CLAIM_SECONDS). A run that passes its deadline with no live claim is ended by the API and fails as “Ran out of time before finishing”.

5. Rejections: a sustained rate of 409 means two workers share one identity; 422 means the worker and API versions do not match, so deploy matching versions.

Caution

Never clear the outbox or edit run records in the database by hand. Cancel a task from the product instead, so the run is fenced and the cancellation event is emitted.

Approvals stuck

Alert: PoloApprovalQueueAged.

1. polo_operations_awaiting_approval counts only requests a person can still decide. With a growing oldest age, notify the workspace’s administrators. Requests counted in polo_operations_undecidable_requests need no action: their task or agent stopped, so they never execute.
2. A decision was taken but nothing happens: if polo_outbox_pending_events is non-zero, follow “Worker stopped”. Otherwise confirm on the task page that the run is still running and the worker is alive.
3. Approve returns 409: the response names the reason. Either the request changed since it was reviewed (refresh and review it again), or the task or the agent that asked has stopped.
4. Approve returns 403: the approver lacks the role, or the reviewed arguments no longer match what the agent would execute. The agent must propose again.
5. An execution that ended as unknown stays that way until verified. Check the runner’s receipt before retrying, so a change that is not idempotent is not repeated.

Approval windows default to 24 hours (POLO_API_APPROVAL_TTL_HOURS). See Approvals and guardrails.

Model route failing

Alert: PoloWorkerUnauthorized, or runs failing immediately with model errors.

1. Identity: a 401 on internal routes means a worker key is not registered in POLO_API_WORKER_KEYS, or a dedicated worker identity is used for the wrong purpose.
2. Route configuration: hosted routes come from POLO_API_HOSTED_MODEL_ROUTES on the API and must match the worker’s model registry entries by id.
3. Provider: the worker’s log names the provider error class (rate limit, authentication, context length). For a slow or local model, confirm that POLO_RUNTIME_ACTIVITY_TIME_FACTOR and POLO_API_CLAIM_SECONDS cover its latency.
4. An agent with a fallback model continues on it, and the task says when it switched. See Models and spend.
5. After a fix, run one small task end to end and confirm it reaches a terminal status.