Troubleshooting

Agent fails at startup or validate

The agent stops before connecting if capability.yaml is invalid or a source cannot be prepared. Run the same checks without touching a running agent:


onprest-agent validate --config /absolute/path/capability.yaml

A configuration error is printed in full. For failures that need more detail, such as a database that cannot be reached or a statement that fails EXPLAIN, the output names a log file (detail_log in JSON output); read that file, not an older one. When the running agent itself fails at startup, the detail is in onprest-agent.log next to the binary. See Agent Local Logs.

Common causes:

The database host or port is unreachable, or the credentials are wrong.
The database user lacks permission for a statement, or for EXPLAIN (on SQL Server, SHOWPLAN).
The SQL does not suit the configured driver.
A secret file has the wrong owner or permissions (on Windows, inherited permissions are refused), or a secret or certificate file is a symbolic link. See Secret files.
A folder root does not exist, or a Wasm module is missing or was not built with onprest-agent code build.
A PostgreSQL environment variable such as PGSERVICE is set in the agent’s environment; see PostgreSQL.

validate does not call HTTP APIs or run any capability, so it cannot catch a wrong upstream endpoint, data-dependent constraint errors, or trigger behavior. Test those with a real call.

Spool problems prevent startup

The spool directory (runtime.spool_dir) must be private to the agent’s user and used by one agent only. Startup fails if another agent holds it, its permissions are wrong, its quota cannot hold the files left from an interrupted save, or a folder recorded for cleanup is unreachable.

Keep the spool and the recorded folders as they are, read the startup message in the local log, fix the access or permission problem, and restart with the same spool path. The agent then finishes the cleanup by itself. Do not edit the spool’s records, and do not delete files in business folders by their .onprest- prefix: the agent removes only the temporary files it can verify and never touches completed files. Move or delete a spool only after a successful start at its original path.

Agent cannot connect

Check that:

gateway.url ends in /ws/agent and uses wss:// for a remote gateway.
The agent host can reach the gateway, and any proxy in between supports WebSocket upgrade.
GATEWAY_AGENT_PUBLIC_KEY matches the agent’s private key.
The agent host’s clock is correct.

doctor checks each step and stops at the first one that fails:


onprest-agent doctor --config /absolute/path/capability.yaml --format json

First failed check	Look at
`config` or `database_*`	`capability.yaml` or database access, as in the first section
`gateway_dns`, `gateway_connect`	Host name, port, DNS, firewall, proxy path
`gateway_tls`	Host name and certificate; the agent uses the OS trust store
`gateway_challenge`, `gateway_verify`	Proxy routing of `/ws/agent/*`, clock, and the key pair

doctor does not replace a running agent’s connection or run a capability.

Capability calls return AGENT_OFFLINE or DEPENDENCY_UNAVAILABLE

AGENT_OFFLINE means no agent is connected, or the gateway has not loaded its contract yet. Check /healthz, then the agent’s stdout, stderr, and local log.

DEPENDENCY_UNAVAILABLE mentioning ONPREST_PUBLIC_BASE_URL on every capability, while /healthz reports agent_connected: true, means at least one capability returns download links and the gateway has no file reference origin. Set it and restart the gateway.

API key is rejected

AUTH_FAILED means the key is missing or wrong:

Send it as Authorization: Bearer ... or X-API-Key.
GATEWAY_API_KEYS_JSON must contain the key_hash printed by create-key, and the gateway must have been restarted since.
The JSON value must be single-quoted when a shell loads gateway.env, or the $ in the hash is expanded.

CAPABILITY_DENIED means the key is valid but its capabilities list does not include the capability.

Rate limited

Three different limits return RATE_LIMITED. The message tells them apart:

Message	Limit	What to check
`too many requests`	Gateway, per source IP	`GATEWAY_RATE_LIMIT_REQUESTS_PER_SECOND`, `GATEWAY_RATE_LIMIT_BURST`, and whether the gateway sees the real client IP behind a proxy
`authentication is busy; retry later`	Gateway, too many new API key checks at once	The number of clients authenticating at the same moment
`capability rate limit exceeded`	Agent, the capability’s `policy.rate_limit`	The capability’s policy or `defaults` and the call rate

None of them ran the capability.

Calls fail with EXECUTION_FAILED

The response is deliberately generic. The real error, such as a database permission error or the upstream API’s status, is only in the agent’s local log.

A call that changes data failed with outcome unknown

Do not retry automatically. Check the real state first, as described in Uncertain outcomes. For LIMIT_EXCEEDED with rolled_back on a mutation, compare policy.max_affected_rows with the statement’s WHERE conditions before raising the limit.

Download links do not work

Links point to the wrong host, or a proxy answers 404: check ONPREST_PUBLIC_BASE_URL and make sure the proxy routes /api/v1/files/ to the gateway.
HTTP 403: use the same API key that made the call.
NOT_FOUND: the link expired (file_ttl), or the gateway or agent restarted or reconnected. Make a new call, keeping in mind what else that call does.
BUSY: another download from the same call is running.
A cut-off file: the transfer failed. Download again while the link is valid.