Keep the host healthy

Readiness that names the failing check, backups that refuse to write plaintext, and upgrades that publish one commit byte for byte.

GET /api/readyz, simulatedsynthetic

Checks: any state but ok answers 503

control_planeThe store opens and answers a read.

schemaThe store has the migrations this code knows.

gateA real gated path is refused without a session.

requestsNo request is past its deadline.

Reported beside the checks, never in them

egress_firewallFirewall coverage of organisation networks.

dns_filterA DNS filter answers on every gateway.

200READY · send it traffic

Every check is ok. Coverage is reported alongside, so an operator can see a gap without the console being taken away from them.

{
  "ready": true,
  "checks": [
    { "check": "control_plane", "state": "ok" },
    { "check": "schema", "state": "ok" },
    { "check": "gate", "state": "ok" },
    { "check": "requests", "state": "ok" }
  ],
  "egress_firewall": { "state": "closed" },
  "dns_filter": { "state": "covered" }
}

Press a state. Any check that is not ok turns the answer to 503, and unknown counts as not ready. The two coverage readings never change the answer.

Health: livez and readyz

Two probes, two questions. /api/livez asks whether the process is running and reads nothing, so it stays 200 while the product is unusable. /api/readyz asks whether the process should get traffic: it reads the store, compares the schema and puts a real request through the gate. It answers 503 and names the failing check. A not-ready process is one to route around while someone reads the sentence, not one to restart on reflex.

on the host

$ curl -s http://127.0.0.1:5000/api/livez
$ curl -s http://127.0.0.1:5000/api/readyz | jq '.ready, .checks'

Probe it without a session. Called from a signed-in browser, readyz can report the gate check as failed. This is an open issue.

What is actually live

A pull updates a checkout; it does not update what is running. These read the host rather than the repository.

bin/whats-running: Whether the live code is the code you pulled, unit by unit, and whether the served console bundle matches the built one. --json for scripts.
bin/deploy status: What is deployed, and whether it matches.
bin/egress-coverage: One sentence per organisation network: covered by the firewall, or not.
bin/store-drift report: Every disagreement between the store and the files that mirror it, as a number.
bin/tenant-audit: Which organisations get more than the default, with a verdict per row.

Backup and restore

bin/backup writes one encrypted part per unit you might need back on its own: the control store, the audit record, the front-door routes, and each workspace's state and home volume. It uses age or gpg, and create refuses until a recipient or passphrase is set. verify decrypts every part, opens the database and re-walks the audit chain; it does not trust the manifest.

Backup checklist00 / 05

Set an age recipient or a gpg key. Without one, create refuses rather than writing plaintext.
Take a backup, with volumes. bin/backup create --volumes
Prove it. bin/backup verify <name> decrypts and counts what landed.
Copy the parts off the host. A backup on the same disk is not a backup.
Add a timer. Nothing schedules bin/backup for you.

on the host

$ bin/backup create --volumes
$ bin/backup list
$ bin/backup verify <name>
# bring back one workspace and touch nothing else
$ bin/backup restore <name> --workspace acme/dev01
$ bin/backup prune --keep 14

Credentials are left out by default. Host credential files are included only with --include-creds. The control store does hold the outbound mail password, so treat every archive as sensitive.
No restore runbook yet. What to back up is documented; a tested disaster-recovery procedure is not. The checklist above is a reading aid, and it stores nothing.

Upgrades and rollback

bin/deploy publishes one commit's files, byte for byte, to the tree every workspace runs. verify exits 0 when the live tree is that commit and 2 when it is not. When the store is newer than the code, a schema-skew guard refuses API calls with a sentence instead of guessing.

Upgrade checklist00 / 06

Take and verify a backup first. See the checklist above.
Read what is live. bin/deploy status
Run the local CI gate on the commit you are about to publish.
Publish it. bin/deploy push --ref <ref> needs sudo; the tree is root-owned.
Prove it. bin/deploy verify --ref <ref>, then readyz.
Re-grant governed runs. A grant is tied to the commit it was proven on, so run bin/deployment-readiness --prepare and grant again.

on the host

$ bin/deploy status
$ bin/deploy push --ref <ref>
$ bin/deploy verify --ref <ref>
# go back to a known commit
$ bin/deploy rollback --ref <ref>

Logs

The ops log is JSON lines with a correlation id, so one request can be followed across every process. It is off until an operator turns it on. The audit record is separate: see Audit & evidence.

on the host

$ bin/obs-log tail
$ bin/obs-log errors --since 1h
$ bin/obs-log query --corr <id>
$ bin/obs-log metrics --since 24h

Capacity

Figures we run onnot a sizing guarantee

Host memory: 8 GB or more
Host disk: 30 GB or more
Per workspace: 2 GB memory · 2 CPUs · 512 processes
Browser IDE: about 1 GB more per person using it
Idle suspend: 30 min, swept every 5 min
Workspaces per host: no figure is stated

Plan about 2 GB of memory for each active workspace. Suspended workspaces keep their volume and give the memory back. See Self-hosted deployment for the host requirements in full.

Troubleshooting

readyz: schema failed

The store has migrations this code does not know, usually after rolling back past an upgrade. Publish the newer commit again; API calls are refused until code and store agree.

readyz: gate failed

First, probe without a session. If an anonymous probe still fails, a gated route answered without a session. Treat it as a security fault and report it.

readyz: requests failed

The answer lists the request that passed its deadline. Route traffic around the process while you read it.

A package install is refused

Ask what the proxy would decide, then read what the organisation actually reached. Members can ask for a host; an admin decides.

on the host

$ bin/egress-proxy check --host registry.example.org
$ bin/egress-proxy summary --days 14

A push is refused

The console shows the broker's refusal with its remedy. On the host, ask the broker directly.

on the host

$ bin/git-broker check --tenant acme --workspace dev01 \
    --ref refs/heads/main

Sign-in fails over plain http

When the host declares TLS, the session cookie is marked Secure and a browser will not send it over http. The login page says so. Use the https address.

An agent run will not start

Open the organisation's “Why it's stuck” tab: the first red gate of five is the reason. The guide walks through it.

What it does not do yet

No metrics endpoint or alerting. Prometheus metrics and alerts are not built. Health is the two probes and the tools above.
Nothing schedules backups. bin/backup exists; you add the timer.
No high availability. The control plane is one SQLite file on one host. Worker hosts are built and off by default.
No workspaces-per-host figure. Size from the per-workspace defaults, and measure.
The ops log is off by default. Until an operator turns it on, there is no correlation trail to query.

Every command Self-hosted deployment

Verified against the main branch, 28 Sep 2026.

Health: livez and readyz

What is actually live

Backup and restore

Upgrades and rollback

Logs

Capacity

Troubleshooting

What it does not do yet

Next